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ABSTRACT 


The  purpose  of  this  study  is  to  provide  eontributing  authors  with  an  understanding 
of  the  trends  in  artiele  submission  to  the  Acquisition  Review  Journal  {ART)  with  regard  to 
the  types  of  researeh  performed,  researeh  design,  and  data  analysis.  This  researeh  will 
provide  future  eontributors  with  insight  that  will  improve  both  the  quality  of  the  ARJ  and 
future  researeh  for  the  Aequisition  Corps.  It  will  also  provide  guidanee  and 
reeommendations  for  future  researeh  artieles  within  the  ARJ. 

This  study  analyzed  and  elassified  233  artieles  that  were  published  in  the  ARJ 
over  the  last  13  years  (1994-2006).  Content  and  statistieal  analyses  were  performed  on 
the  themes,  research  types,  research  designs,  and  data  analysis  methods  employed. 
Moreover,  trends  such  as  educational  and  institutional  affiliations  of  contributing  authors 
were  also  reviewed. 

The  ARJ  has  shown  some  distinctive  trends,  which  are  reflected  in  its  publication 
of  a  number  of  qualitative  studies;  however,  it  has  also  shown  progress  in  the  number  of 
published  quantitative  studies.  The  academe  and  practitioners’  contributions  remained 
steady,  while  civilian  contributions  have  been  rising.  These  trends  are  in  line  with 
current  recommendations  of  the  research  community.  For  a  relatively  new  journal,  such 
trends  are  a  good  representation  of  a  growing  community. 
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I.  INTRODUCTION 


A,  BACKGROUND 

1,  History  of  Acquisition  Reform 

In  1961,  Secretary  of  Defense  Robert  S.  McNamara  attempted  to  revamp  the 
procurement  system.  i  From  1961  on,  several  acquisition  reform  initiatives  followed  as 
shown  in  Figure  1. 


Year 

Initiative 

1961 

McNamara  initiatives 

1970 

Fitzhugh  Commission 

1972 

Commission  on  Government  Procurement 

1976 

0MB  Circular  A-109 

1978 

Defense  Science  Board  Acquisition  Cycle  Study 

1979 

Defense  Resources  Management  Study 

1981 

Defense  Acquisition  Improvement  Program 

1983 

Grace  Commission 

1986 

Packard  Commission 

1986 

Goldwater  Nichols 

1989 

Defense  Management  Review 

Figure  1 .  Reform  Initiatives^ 

These  reform  efforts  emphasized  streamlining  the  weapons  acquisition  process, 
improving  cost  estimating  practices,  and  changing  personnel  procedures  to  produce  more 
qualified  contracting  staff.  Recommendations  have  included: 

•  eliminating  needless  legal  encumbrances  on  contracting  procedures; 

•  empowering  program  managers; 

•  establishing  clear  lines  of  authority; 

•  simplifying  the  source  selection  process; 


1  Deborah  Frank,  “A  Theoretical  Consideration  of  Acquisition  Reform,”  Acquisition  Review 
Quarterly,  Summer  1997,  p.  281. 

2  Ibid. 


1 


•  reducing  technical  criteria; 

•  re-codifying  federal  laws  governing  procurement; 

•  employing  more  frequent  product  testing  and  competitive  prototyping; 

•  improving  the  pay,  training,  and  career  options  for  personnel;  and 

•  multi-year  congressional  funding. 3 

The  efforts  mentioned  above  are  themes  that  have  been  expressed  throughout  the 
articles  and  issues  of  the  Acquisition  Review  Journal  {ART).  The  ARJ  was  intended  for 
the  following  purpose: 

.  .  .  Acquisition  Corps  members  and  other  readers  from 
government.  Congress,  industry  and  academe  are  encouraged  to  use  the 
Acquisition  Review  Quarterly  {ARQ),  now  ARJ,  as  their  professional 
forum  for  discussion  and  exchange  of  policies,  research,  information, 
and  opinion.^ 

2,  Relevance  of  the  ARJ 

The  ARJ  encourages  articles  on  policy,  scholarly  research,  opinions,  and  anything 
else  that  affects  the  acquisition  community.  Acquisition  is  a  very  broad  term,  which  is 
why  the  ARJ  welcomes  articles  in  all  areas  such  as  contracting,  logistics,  management, 
and  technology.  For  example,  in  February  2006,  Lieutenant  General  Donald  J.  Hoffman, 
United  States  Air  Force  (Military  Deputy,  Office  of  the  Assistant  Secretary  of  the 
Air  Force  for  Acquisition)  stated  the  following:  “I  stand  by  my  earlier  comments  on  the 
quality  of  our  military  hardware,  but  you  have  all  read  the  headlines  about  acquisition 
reform  and  the  biggest  driver  is  the  need  to  control  cost  and  schedule. ”3  His  viewpoint  is 
reflected  in  this  study’s  results  as  Cost,  Schedule,  and  Performance  is  the  second  highest 
theme  category  of  the  ARJ,  with  Technology  being  number  one. 


3  Lauren  Holland,  “The  Weapons  Acquisition  Process:  The  Impediments  to  Radical  Reform,” 
Acquisition  Review  Quarterly,  Spring  1998,  p.  235. 

^Acquisition  Review  Quarterly,  Editorial  Mission,  Summer  1994,  no  page  number  noted. 

3  Lieutenant  General  Donald  J.  Hoffman,  United  States  Air  Force,  “The  Way  Ahead,”  February  2006, 
https://www.safaa.ha.afmil/news/februarv2Q06/hoffman.html.  last  accessed  on  29  November  2006. 
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B,  HISTORY  OF  THE  ARJ 

One  of  the  two  publieations  by  the  Defense  Aequisition  University  (DAU),  aside 
from  the  Acquisition,  Technology,  and  Logistics  (AT&L)  magazine,  is  the  ARJ,  whieh  has 
eonsistently  published  artieles  for  both  aeademe  and  praetitioners  with  a  partieular 
interest  in  the  aequisition  eommunity.  During  its  13-year  life  span,  it  has  published 
42  issues  and  233  artieles,  not  ineluding  editorial  and  speeial  edition  introduetions  and 
books  reviews.  A  total  of  325  authors  have  published  artieles  in  the  ARJ  (see 
Appendix  A).  The  ARQ  was  renamed  the  ARJ  for  Issue  35  (2004),  eoneurrent  with  a 
deerease  in  the  number  of  issues  published  per  year  from  four  to  three.  Although  this 
publieation  experieneed  a  name  ehange,  the  eontent,  intent,  and  mission  did  not  ehange. 
The  ARJ  was  designed  to  address  the  needs  of  professionals  aeross  the  defense 
aequisition  speetrum.  Its  editorial  mission  is  to: 

.  .  .  provide  praeticing  aequisition  professionals  with  relevant 
management  tools  and  information  based  on  reeent  advanees  in  policy, 
management  theory,  and  research.  The  ARQ  {ARJ)  addresses  the  needs  of 
professionals  across  the  full  spectrum  of  defense  acquisition,  and  is  intended 
to  serve  as  the  mechanism  for  fostering  and  disseminating  scholarly  research 
on  acquisition  issues,  for  exchanging  opinions,  for  communicating  policy 
decisions,  and  for  maintaining  a  high  level  of  awareness  regarding 
acquisition  management  philosophies.  In  addition  to  the  acquisition 
professional,  the  ARQ  {ART)  provides  insight  to  others  in  the  Department  of 
Defense  (DoD),  Congress,  industry  and  academe  who  have  significant 
interest  in  how  the  DoD  conducts  its  acquisition  mission.  Acquisition  Corps 
members  and  other  readers  from  government.  Congress,  industry  and 
academe  are  encouraged  to  use  the  ARQ  {ART)  as  their  professional  forum 
for  discussion  and  exchange  of  policies,  research  information, 
and  opinions.^ 

Based  on  the  editorial  mission  statement,  the  ARJ  should  serve  as  a  forum  for  the 
exchange  of  opinions,  communicating  policy  decisions,  and  for  maintaining  awareness 
for  acquisition  management  philosophies.  Not  only  should  it  be  a  forum  for  the 
community,  but  it  should  also  be  used  as  a  means  for  communication,  both  for  the 
members  of  the  acquisition  community,  and  to  all  those  that  have  an  expressed  interest  in 


^  Acquisition  Review  Quarterly,  Editorial  Mission,  Summer  1994,  no  page  number  noted. 
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the  DoD’s  acquisition  mission.^  Members  of  the  Acquisition  Corps  and  other 
communities  (e.g.,  logistics,  contracting,  end  user,  etc.)  are  encouraged  to  use  the  ARJ  as 
a  professional  platform  for  information  exchange  among  community  members. 
Gerald  E.  Knightley,  the  Executive  Director  at  the  time  of  its  inception,  believes  that  the 
ARTs  target  audience  is  senior  members  of  the  Acquisition  Corps.* * 

The  y4i?Jhas  had  four  editors  over  its  13 -year  existence  (see  Table  1).  In  Table  1, 
Norene  Taylor  is  referred  to  as  an  editor;  however,  in  Issue  32  (published  in  summer 
2003)  the  position  of  editor  was  disseminated  among  three  positions  and  she  was 
designated  as  Managing  Editor  (lead  editor). 


Time  Period 

Issues 

Editor 

Winter  1994-Spring  1995 

5 

Robert  W.  Ball 

Summer  1 995 -E all  1997 

6 

James  Kurt  Wittmeyer 

Winter  1997-Summer  2000 

10 

Deborah  L.  Gonzalez 

Eall  2000-Present  (2006) 

21 

Norene  E.  Taylor 

(Formally  Blaneh  and  Fagan-Blaneh) 

Table  1.  Editors 


At  the  end  of  each  issue  of  the  ARJ,  there  is  a  section  called  “Guidelines  for 
Authors,”  which  specifies  the  ARfs  criteria  for  manuscript  submission.  In  the  1994 
winter  edition,  for  example,  the  ARJ  was  interested  in  articles  that 

.  .  .  represent  scholarly  examination,  disciplined  research,  and 
supported  empirical  experience  in  the  fields  of  defense  systems 
management  and  acquisition  management.  .  .  .  Manuscripts  supporting  the 
Defense  Acquisition  University  (DAU)  commitment  to  improve  the 
acquisition  process  and  the  professionalism  of  the  acquisition  workforce 
are  particularly  welcome.^ 

While  articles  related  to  acquisition  were  preferred,  other  fields  of  management 
would  not  be  omitted. 


^  Gerard  E.  Knightley,  “Aequisition  Review  Quarterly,”  Memorandum  for  Defense  Aequisition 
Professionals,  Acquisition  Review  Quarterly,  Winter  1994. 

*  Ibid. 

^  Acquisition  Review  Journal,  “Guidelines  for  Authors,”  Winter  1994,  p.  84. 
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The  summer  1995  issue  experieneed  a  eover  change,  but  it  also  had  an  addendum 
to  the  “Guidelines  for  Author”  section  in  regard  to  the  types  of  manuscripts  it  desired  to 
publish:  “The  ARJ  welcomes  manuscripts  from  anyone  interested  in  the  defense 
acquisition  process.”!*’  Further  change  occurred  in  the  winter  1997  issue,  which 
explicitly  defined  defense  acquisition  “.  .  .  as  the  conceptualization,  initiation,  design, 
development,  logistics  support,  modification,  and  disposal  of  weapons  and  other  systems, 
supplies,  or  services  to  satisfy  Defense  Department  needs,  or  intended  for  use  in  military 
missions.”!!  The  next  noticeable  change  was  in  the  summer  1999  issue,  in  which  authors 
were  asked  to  include  a  photograph  of  themselves  and  their  email  address,  to  be  placed 
with  their  biography.  In  Issue  37,  in  the  “Guidelines  for  Authors”  section,  the  ^47?/ began 
to  publish  planned  issue  themes  and  publication  schedules.  In  this  section,  they  displayed 
the  due  date  and  publication  date;  however,  in  order  to  learn  the  themes,  prospective 
authors  needed  to  view  the  DAU’s  Website  (http://www.dau.mil).  Another  change,  made 
in  Issue  39,  added  themes  to  the  “Guidelines  for  Authors”  section,  right  under  the  due 
date  and  publication  date. 

Over  the  years,  the  ARJ  evolved  in  order  to  better  serve  the  acquisition 
community.  Chapter  11  begins  to  explore  the  trends,  categories,  and  themes  that  were 
gathered  from  reading  the  articles  that  have  been  published  since  its  inception. 

C.  PURPOSE  OF  STUDY 

1.  Research  Statement 

Prior  to  2005,  a  study  of  the  ARJ  had  not  been  conducted.  However,  in  2005, 
Elder  did  a  thesis  on  the  ARJ.  In  his  study,  he  reviewed  all  the  articles  published  in  the 
ARJ  up  to  Issue  36  (released  in  2004),  which  provided  him  with  a  total  of  193  articles 
after  he  excluded  editorial  and  special  edition  introductions  and  book  reviews.  This  study 
will  include  the  articles  that  Elder  reviewed  plus  the  remaining  ones  to  date,  providing  us 
with  a  grand  total  of  233  articles. 


!*!  Acquisition  Review  Journal,  “Guidelines  for  Authors,”  Summer  1995,  baek  eover. 
!!  Acquisition  Review  Journal,  “Guidelines  for  Authors,”  Winter  1997,  p.  127. 
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The  purpose  of  this  study  is  to  provide  eontributing  authors  with  an  understanding 
of  the  trends  in  artiele  submission  to  the  Acquisition  Review  Journal  {ART)  with  regard  to 
the  types  of  researeh  performed,  researeh  design,  and  data  analysis.  This  researeh  will 
provide  future  eontributors  with  insight  that  will  improve  both  the  quality  of  the  ARJ  and 
future  researeh  for  the  Aequisition  Corps.  It  will  also  provide  guidanee  and 
reeommendations  for  future  research  articles  within  the  ARJ. 

The  intentions  of  this  study  are:  (1)  to  replicate  Elder’s  thesis  and  compare  the 
two  sets  of  results;  (2)  to  analyze  the  remaining  articles  to  date  and  determine  if  there  has 
been  any  noticeable  changes  in  trends  in  the  types  of  articles,  the  themes,  and  the 
characteristics  of  the  contributing  authors;  and  (3)  to  analyze  these  trends  over  the  past 
13  years  within  the  acquisition  community.  Replicating  Elder’s  thesis  and  conducting  a 
second  content  analysis  of  the  ARJ  will  provide  confirmation  of  its  focus  on  Acquisition 
Reform  and  if  it  truly  provides  a  forum  for  the  acquisition  community  as  described  in  the 
ARJ  mission. 

Other  research  communities  have  performed  studies  on  research  that  has  been 
done  in  their  respective  communities.  An  example  of  this  type  of  study  was  performed 
by  two  Supply  Chain  Management  professors,  Craig  R.  Carter  and  Eisa  M.  Ellram,  who 
published  the  article,  “Thirty-Eive  Years  of  the  Journal  of  Supply  Chain  Management: 
Where  Have  We  Been  and  Where  are  We  Going?”  In  their  study,  they  reviewed  all  the 
articles  within  the  Journal  of  Supply  Chain  Management  and  noted  the 
following  information: 

•  The  editors  over  the  existence  of  the  journal 

•  The  authors 

o  Their  educational  background 

o  Their  institutional  affiliation 

•  The  article’s  subject  categories 

•  The  type  of  research  performed 

•  The  type  of  research  design 

•  The  type  of  data  analysis 
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Once  all  the  information  was  colleeted,  they  eompleted  a  eontent  analysis  of  the 
data  and  noted  any  trends  that  existed.  Their  researeh  was  well  designed  and  so 
eomprehensive  that  the  Acquisition  Community  felt  it  would  be  both  interesting  and 
beneficial  to  conduct  a  similar  study  on  the  ARJ. 

This  study  is  a  replication  of  the  eontent  analysis  of  the  ARJ  using  two  coders 
(researehers).  The  two  coders  have  the  same  professional  background  and  educational 
experience.  The  present  endeavor  eompares  the  results  of  the  two  independent  studies. 
In  addition,  the  two  eoders  separately  analyzed  the  articles  published  in  the  ARJ  through 
2006.  A  third  eoder  was  used  as  an  arbitrator  to  resolve  any  differences  in  analysis. 

2,  Research  Questions 

Having  read  the  ART s  editorial  mission,  the  following  questions  were  structured. 

•  How  does  the  ARJ  contribute  to  the  aequisition  community  as  a  forum  for 
intellectual  exchange? 

•  Who  is  publishing  the  articles  in  the  ARJl 

•  What  determines  the  caliber  of  the  artieles  published  and  why  are  those 

artieles  ehosen  over  others? 

3.  Investigative  Questions 

Based  on  Carter  and  Ellram’s  content  analysis  methodologies,  the  coders 
developed  similar  research  questions  to  guide  their  primary  researeh. 

•  What  subject  matters  or  themes  have  been  addressed  in  the  ARJl 

o  Do  the  subjeet  matters  or  themes  enable  the  readers  to  gain  new 
insight  about  a  partieular  aequisition  issue;  develop  new  concepts 
or  theoretieal  perspectives  about  aequisition;  or  discover  problems 
and/or  solutions  to  particular  issues? 
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•  How  have  the  themes  been  explored  or  written  in  the  past  13  years? 

o  What  researeh  methodologies  have  been  employed  in  the  journal? 
What  is  the  most  commonly  employed  research  methodology? 

o  What  type  of  data  collection  and  data  analysis  did  the  authors  use 
to  support  their  research/contribution?  From  among  the  different 
types  of  data  analysis,  which  appears  to  be  most  common? 

o  Are  there  trends  in  the  type  of  articles,  research  categories,  or  data 
analysis  over  the  years?  Do  the  types  differ  across  time? 

o  What  does  this  analysis  suggest  about  the  study  of  acquisition 
reform  in  the  ARJl 

•  What  types  of  authors  have  contributed  articles  to  the  ARJl 

o  Are  they  academe  or  practitioners? 

o  What  are  their  institutional  affiliations?  Do  the  authors  come  from 
military  or  civilian  institutions? 

o  What  is  the  trend  on  the  number  of  authors  per  article  over  the 
years?  Did  the  number  of  authors  per  article  change  across  time? 
Are  the  authors  encouraged  to  collaborate  with  other  authors? 

o  Do  the  articles  published  to  this  point  appear  relevant  with 
on-going  acquisition  reform? 

D.  ORGANIZATION  OF  RESEARCH 

The  study  is  organized  into  four  chapters.  Chapter  I  provides  an  introduction  and 
overview  of  the  study.  Chapter  II  describes  the  methodology,  including  the  processes 
and  techniques  used  to  acquire  and  analyze  the  data.  Chapter  III  provides  an  in-depth 
analysis  of  the  data,  including  the  results  from  the  replication  of  Elder’s  study.  Finally, 
Chapter  IV  summarizes  the  conclusions  and  provides  recommendations  for  future  study. 
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II.  RESEARCH  METHODOLOGY 


The  study  focused  on  classifying  233  articles  that  were  published  in  the  ARJ  oyqx 
the  last  13  years.  The  coders  read  all  the  articles  published  since  the  journal’s  inception 
in  1994,  with  the  exception  of  editorial  and  special  edition  introductions  and  book 
reviews.  In  this  chapter,  we  will  discuss  the  methodology  used  to  classify  the  articles  in 
order  to  discuss  different  aspects  of  the  ARJ.  The  methodology  includes  types  of 
research  performed,  types  of  research  design,  and  types  of  data  analysis,  statistical 
methods,  and  themes. 

A,  DATA  COLLECTION 

To  gather  all  the  data  required  for  the  analysis,  the  coders  first  created  an  Excel 
spreadsheet  to  capture  all  the  areas  relevant  to  achieving  the  answers  to  the  research 
questions.  The  coders  performed  a  content  analysis  of  the  themes,  research  types, 
research  designs,  and  data  analysis  methods  employed.  A  number  of  other  authors  have 
conducted  studies  of  this  stature  within  their  respective  communities  such  as: 

•  Carter  and  Ellram 

•  Das  and  Hanfield 

•  Williams  and  Oumlil 

•  Mentzer  and  Kahn 

The  study  performed  a  content  analysis  of  the  themes,  research  types,  research 
designs,  and  data  analysis  methods  employed. 

In  order  to  collect  the  data,  the  process  required  both  coders  to  read  articles 
independently,  while  simultaneously  filling  in  two  data  spreadsheets.  Upon  completion 
of  the  articles  published  in  1994  and  1995,  a  meeting  of  the  coders  was  held  to  compare 
results.  The  reason  for  the  meeting  was  to  ensure  that  both  individuals  had  similar 
interpretations  of  the  definitions  they  applied  to  the  articles,  respectively.  Each  article 
was  discussed  individually  and  an  exchange  of  the  themes  and  categories  took  place.  The 
articles  on  which  the  coders  were  in  100  percent  agreement  were  noted  and  upon 
completion  of  all  233  articles,  a  reliability  rate  was  calculated.  The  reliability  rate 

9 


consists  of  the  total  number  of  articles  that  were  in  100  percent  agreement  divided  by  the 
total  number  of  articles  (233),  which  resulted  in  a  91  percent  agreement  rate.  This 
percentage  is  known  as  the  inter-coder  agreement,  in  which  an  agreement  rate  of 
70  percenti2  or  higher  is  favorable.  For  those  artieles  about  whieh  there  was  not 
100  percent  agreement,  a  third  coder  was  used  to  settle  any  discrepancies.  The  purpose 
of  the  third  coder  was  to  read  the  articles  and  perform  the  same  assessment,  which  was 
used  to  adjudieate  any  disagreements.  For  instance,  if  the  first  coder  assigned  a 
Normative  review,  while  the  second  assigned  a  Literature  review,  the  disagreement  was 
broken  by  the  categorization  of  the  third  coder.  The  importance  of  the  third  eoder  was  to 
ensure  that  a  one-to-one  ratio  was  maintained  for  each  category  per  article. 

In  the  beginning,  the  eoder  meetings  were  critically  important  since  the 
definitions  of  the  categories  were  being  refined  for  proper  interpretation  by  both 
individuals.  According  to  Rust  and  Cooil,  reliability  can  be  improved  in  many  ways. 
One  good  option  is  to  add  more  coders,  but  this  option  is  not  always  feasible.  Another 
option,  to  make  it  more  praetical  and  to  improve  the  quality  of  the  judging  itself,  is  to 
make  the  instructions  clearer  and/or  improve  the  methodology.  Additionally,  better 
reliability  can  be  achieved  through  inter-coder  agreement;  here  the  coders  provide 
information  and  rationale  rather  than  random  guesses.  The  coders  focused  on  following 
a  clear  methodology,  which  led  to  high  inter-eoder  agreement. 

In  order  to  perform  the  categorical  and  statistical  analyses,  both  researchers  were 
responsible  for  counting  the  number  of  eategories  chosen  for  each  research  type  within 
their  own  spreadsheet.  Each  coder  used  a  different  method  to  conduct  a  count  of  the 
categories:  one  coder  used  Excel  to  sort  and  eount  (histogram  and  count-if  functions), 
while  the  other  coder  counted  manually.  The  two  methods  derived  the  exaet  same 
results.  Afterward,  the  two  eoders  compared  their  eount  results  to  see  if  they  were  both 
in  agreement,  then  they  calculated  the  frequencies  for  eaeh  of  the  three  types. 


12  John  C.  Nunnally  and  Ira  Bernstein,  Psychometric  Theory,  McGraw  Hill,  1994,  p.  265. 

12  Roland  T.  Rust  and  Bruce  Cooil,  “Reliability  Measures  for  Qualitative  Data:  Theory  and 
Implications,”  Jowma/  of  Marketing,  February  1994,  p.  2. 
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B,  TYPE  OF  RESEARCH  PERFORMED 

The  type  of  researeh  performed  was  one  of  the  three  areas  identified  for  each  of 
the  233  articles.  In  order  to  classify  each  article,  the  coders  used  Mentzer  and  Khan’s 
topology.  The  types  and  the  definitions  that  were  used  to  categorize  each  article  are 
displayed  in  Table  2.  The  frequency  column  in  Tables  2,  3,  and  4  displays  the  percentage 
of  times  that  the  respective  category  was  chosen  among  the  233  articles.  The  categories 
were  noted  and  tabulated,  after  all  disagreements  were  settled,  and  then  the  total  number 
of  articles  per  category  was  divided  by  233  (the  total  number  of  articles).  The  frequency 
tells  the  community  the  number  of  times  that  a  particular  category  has  been  used  in 
the  ARJ. 

The  agreement  rate  between  the  two  coders  was  96  percent — considerably  above 
the  70  percent  minimum  inter-coder  agreement  rate  and  better  than  the  85  percent 
coefficient  of  agreements!^  recommended  by  Harold  Kassarjian  in  1977.  As  mentioned 
in  Section  A,  a  third  coder  and  meetings  were  used  to  adjudicate  any  disagreements. 


14  John  Mentzer  and  Kenneth  A.  Khan,  “A  Framework  of  Logistics  Research,”  Journal  of  Business 
Logistics,  Vol.  16,  No.  1,  1995,  p.  241. 

1^  Flarold  FI.  Kassarjian,  “Content  Analysis  in  Consumer  Research,”  Journal  of  Consumer  Research, 
Vol.  4,  No.  1,  1997,  pp.  8-18. 
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Category 

Definition 

Frequency 

Normative 

Review 

“Research  that  examines  what  ought  to  be  and  what  individuals 
and  organizations  ought  to  do.”i®  Literature  might  be  cited  in 
the  article,  but  the  point  of  the  inclusion  of  this  literature  is  to 
support  the  opinions/assertions  of  the  author.  Normative 
research  aims  at  improvements,  which  means  that  it  includes 
evaluation  of  the  present  state  of  things  and  also  of  the 
direction  of  future  development. 

28.3% 

Literature 

Review 

This  type  is  a  review  and  synthesis  of  existing  literature,  the 
result  of  which  is  the  development  of  a  framework, 
propositions,  or  normative  prescriptions  grounded  on 
existing  literature. 

19.3% 

Exploratory 

Study 

This  research  makes  observation  of  [Acquisition]  for  the 
purposes  of  developing  theories,  but  leaves  the  testing  of  the 
theories  for  other  studies. ”^7  It  means  that  the  researcher  starts 
by  gathering  as  much  information  about  the  [subject]  and 
jotting  everything  down  until  he  gets  a  better  picture  of  what  is 
necessary  (theory).  There  is  no  formal  hypothesis,  its  purpose 
is  to  explore  some  areas  more  thoroughly  to  develop  some 
specific  hypothesis  or  prediction  than  be  tested  in  future 
research. 

26.6% 

Methodology 

It  is  a  review  of  research  methodologies  used  in  the  field  of 
[Acquisition],  typically  in  “how-to”  form.  It  introduces  an 
academic  research  methodology  as  well  as  a  practitioner 
methodology. 

15.9% 

Hypothesis 

Testing 

When  a  researcher  wants  to  test,  the  researcher  starts  with  a 
hypothesis.  A  hypothesis  is  a  specific  statement  of  predictioni^ 

9.9% 

Table  2.  Type  of  Researeh  Performed^o 

The  most  eornmon  type  of  research  performed  was  Normative  review. 
“Civil-Military  Integration;  The  Context  and  Urgeney”  (Article  #102,  Appendix  A)  by 


John  Mentzer  and  Kenneth  A.  Kahn,  “A  Framework  of  Logistics  Research,”  Journal  of  Business 
Logistics,  Vol.  16,  No.  1,  1995,  pp.  240-241. 

17  Ibid. 

1^  William  M.  K.  Trochim,  “Deductive  and  Inductive  Thinking,”  http://www.socialresearchmethod. 
net/kb/dedind.htm,  20  October  2006,  last  accessed  on  12  December  2006. 

m  William  M.  K.  Trochim,  “Hypotheses,”  http://www.socialresearchmethod.net/kb/hypothes.htm, 
20  October  2006,  last  accessed  on  12  December  2006. 

Craig  Carter  and  Lisa  Ellram,  “Thirty  Five  Years  of  the  Journal  of  Supply  Chain  Management: 
Where  Have  We  Been  and  Where  are  We  Going?,”  Journal  of  Supply  Chain  Management,  Spring  2003, 
p.  31. 
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William  B.  Scott,  which  talked  about  the  need  for  a  strong  industrial  base  to  maintain  our 
eeonomie  and  military  strength,  is  an  example  of  Normative  review.  A  thin  line  appears 
to  exist  between  Normative  and  Literature  reviews.  In  the  ease  of 
Dr.  William  Washington,  the  eoders  elassified  some  of  his  artieles  as  Normative  review, 
while  others  were  elassified  as  Literature  review.  With  eontraeting  as  the  main  theme  for 
all  of  his  eontributions.  Dr  Washington  performed  Literature  reviews  on  the  eomparison 
of  Competition  and  Sole  Souree  Proeurement  proeesses  (Artieles  #50,  Appendix  A)  and 
Subeontraeting  (Artiele  #107,  Appendix  A).  His  Normative  eontributions  inelude  artieles 
on  approaehes  to  “Reward  Contracting”  (Article  #57,  Appendix  A)  and  “Partieipatory 
Contraeting”  (Artiele  #1 14,  Appendix  A). 

Only  1.3  percent  separated  Normative  Literature  from  Exploratory  Studies. 
Mr.  Riehard  Sylvester  and  Mr.  Joseph  Ferrara’s  contribution  in  the  winter  2003  issue, 
“Confliet  and  Ambiguity  Implementing  Aequisition,”  explored  the  struggles  of 
implementation  due  to  policy  ambiguity  that  drove  organizational  eonfliet  (Artiele  #164, 
Appendix  A).  In  Table  2,  Hypothesis  Testing  was  the  least  common  research  type. 
Examples  inelude  studies  on  eost  overruns,  estimate  at  eompletion  (EAC)  diseussions, 
lean  strategies,  and  aequisition  reform  initiatives. 

C.  TYPES  OF  RESEARCH  DESIGN 

In  Table  3,  the  types  of  research  designs  identified  in  this  study  are  noted,  along 
with  their  respeetive  frequency  of  usage  The  agreement  rate  for  this  seetion  was 
96  pereent,  with  the  use  of  a  third  eoder  and  diseussion  to  settle  any  disagreements. 
Based  on  the  frequeney,  the  majority  of  the  artieles  were  eategorized  as  Case  Studies. 
Some  authors  did  Case  Studies  on  partieular  systems  sueh  as  Joint  Stars  (Airborne 
Standoff  Target  Aequisition  Reeognition  System),2i  Ml  02  Howitzers,22 
Naval  Air  Systems  Command  (NAVAIR)  team  and  F/A-18  integrated  produet  teams 


21  Article  #12,  “Coming  Up  Golden:  Defense  Acquisition  Board  Review  Guide  for  Program  Offices,” 
Appendix  A. 

22  Article  #30,  “The  Impact  of  Technical  Data  Transfers  Problems  during  a  Transition  of  Weapons 
System  Production  between  Nations,”  Appendix  A. 
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(IPX), 23  the  Defense  Contract  Management  Command  (DCMC)24  in  Scaly,  Texas,  the 
DAU  consortium,25  and  Marine  Corps  Maintenance  Center .26  Not  far  behind 
Case  Studies  were  Topic  Presentations.  Experiments  and  Simulation,  as  well  as 
Mathematical  Models,  were  tied  for  the  least  common  type  of  research  design.  In 
“Acquisition  Reform  Theory  and  Experimental  Evidence  for  Tournament  Sponsors,”  the 
four  authors  performed  an  experiment  suggesting  that  carefully  designed  research 
tournaments  can  be  highly  effective  at  promoting  research  efforts. 22  Dr.  Mark  Nissen, 
the  author  of  “Reengineering  the  REP  Process  through  Knowledge-Based  Systems,” 
illustrated  “the  use  and  utility  of  a  knowledge-based  systems  to  support  process  redesign 
are  demonstrated,  and  insight  is  provided  into  the  potential  of  [Artificial  Intelligence] 
(Al)-based  technologies  to  dramatically  improve  military  procurement.”28  The  last 
category  in  this  section,  the  Mathematical  Model,  was  represented  by  two  authors  who 
co-wrote  two  sequels  to  a  specific  issue  on  risk.  In  2003,  Mr.  Paul  Garvey  and 
Mr.  Chien-Ching  Cho  wrote  an  article  titled  “An  Index  to  Measure  a  System’s 
Performance  Risk”  (Article  174,  Appendix  A)  and  in  2006,  it  was  followed  by  “An  Index 
to  Measure  a  System-of-System’s  Performance  Risk”  (Article  225,  Appendix  A).  Both 
mathematically-oriented,  these  two  articles  showed  computations  of  the  technical  risk 
index  (TRI)  to  improve  technical  performance  by  lessening  risk.  Additionally,  the 
articles  were  almost  identical,  except  that  the  second  article  added  focus  on 
system-of-systems  (SoS). 


23  Article  #72,  “Team-Based  Redesign  as  Large-Scale  Change:  Applying  Theory  to  the 
Implementation  of  Integrated  Product  Teams,”  Appendix  A. 

24  Article  #77,  “Concept  of  Operations  and  Implementation  Plan  for  Industry  Integrated  Logistics 
System  (I2LS),”  Appendix  A. 

23  Article  #83,  “A  Case  Study  for  the  Systems  Approach  for  Developing  Curricula:  ‘Don’t  Throw  Out 
the  Baby  with  the  Bath  Water,’”  Appendix  A. 

26  Article  #187,  “Applying  Theory  of  Constraints  Principles  and  Lean  Thinking  at  the  Marine  Corps 
Maintenance  Center,”  Appendix  A. 

22  Article  #95,  “Acquisition  Reform  Theory  and  Experimental  Evidence  for  Tournament  Sponsors,” 
Appendix  A. 

2^  Article  #41,  “Reengineering  the  RFP  Process  Through  Knowledge-Based  Systems,”  Appendix  A. 
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Category 

Definition 

Frequency 

Archival  Data 

Data  that  already  exist  and  that  have  been  collected  by 
others.  This  archival  data  is  the  simplest  kind  of  data  to 
gather  because  someone  else  has  already  done  the  work. 

9.9% 

Case  Study 

In-depth  data  are  gathered  pertaining  to  a  program  or  event. 

39.9% 

Experiment  and 
Simulation 

Data  collected  through  an  experimental  process  or  through 
a  simulation  process.  Due  to  the  infrequent  occurrences, 
we  combine  them  into  one  category. 

3.9% 

Interviews 

Data  collected  through  conduct  of  interviews. 

9.9% 

Mathematical  Model 

Data  collected  and  explained  using  a  mathematical  model. 

3.9% 

Surveys 

Data  collected  through  surveys. 

7.3% 

Topic  Presentation 

No  discernable  design  methodology. 

25.3% 

Table  3.  Types  of  Research  Design^^ 

These  results  indicate  that  of  the  233  articles,  approximately  60  percent  have  been 
qualitative  studies  and  not  quantitative-  a  figure  that  agrees  with  Elder’s  study. 


D.  TYPES  OF  DATA  ANALYSIS 


Table  4  has  a  list  of  the  types  of  data  analysis  that  the  coders  used  to  describe 
each  of  the  articles.  The  coders  were  in  agreement  92  percent  of  the  time.  Again,  a  third 
coder  was  used  to  assist  with  any  disagreements  between  the  primary  coders. 


Mitchell  J.  Elder,  Capt,  USAF,  “An  Eleven-Year  Retrospective  of  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  2005,  p.  12. 
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Category 

Definition 

Frequency 

Anecdotal 

Evidence 

It  is  an  informal  account  of  evidence  in  the  form  of  an 
anecdote  or  hearsay.  It  is  based  on  incidental  observations 
on  reports  rather  than  on  systematic  evaluation.  The 
argument  draws  a  conclusion  from  cases  specifically 
chosen  to  support  the  conclusion. 

10.7% 

Comparison 

Analysis 

This  type  compares  two  or  more  things  (e.g.,  DoD  versus 
commercial  organizations). 

29.2% 

Content 

Analysis 

When  we  say  content  analysis,  the  researchers  use  detailed 
systematic  evaluation  of  a  particular  body  of  material  for 
the  purpose  of  identifying  patterns,  themes,  or  biases.30 

35.2% 

No  Analysis 

No  particular  analysis  for  the  article. 

15.5% 

Statistical 

Analysis 

To  analyze  the  data,  researches  use  statistical  models  and 
techniques.  This  category  may  include  correlation, 

regression,  descriptive  statistics,  and  means  testing. 

9.4% 

Table  4.  Types  of  Data  Analysis^i 

Content  Analysis  was  the  most  frequent  type  of  analysis  at  35.2  percent,  and  is 
apparent  in  several  articles  in  Appendix  A.  Comparison  and  Content  Analysis  comprise 
approximately  65  percent  of  articles  that  were  published  in  the  ARJ.  This  again  suggests 
that  this  study  is  still  performing  more  qualitative,  rather  than  quantitative,  analysis. 
Statistical  Analysis  is  the  least  preferred  method  noticed  in  the  ARJ.  Statistical  Analysis 
includes  various  techniques  such  as  Analysis  of  Variance  (ANOVA),  empirical  analysis, 
descriptive  statistics,  and  means  testing.  ANOVA  was  employed  by  the  authors  of 
“Investigating  the  Integration  of  Acquired  Firms  in  High-Technology  Industries”  (Article 
#178,  Appendix  A)  “to  identify  the  significance  and  individual  level  of  variance  of  the 
different  variables  based  on  a  firm’s  environment  and  the  timing  of  acquisition.”^^  A 
Descriptive  Statistics  example  is  the  article  on  the  “Relationship  of  Cost  Growth  and 
Schedule  Growth”  (Article  169,  Appendix  A),  where  the  data  points  from  the  RAND 
Cost  Growth,  Selected  Acquisition  Report  (SAR)  were  analyzed  using  the  mean,  standard 

30  Mitchell  J.  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  2005,  p.  13. 

31  Ibid.,  p.  14. 

32  John  Driessnack  and  David  King,  “Investigating  the  Integration  of  Acquired  Firms  in 
High-Technology  Industries,”  Acquisition  Review  Journal,  Summer  2003,  p.  269. 
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deviation,  coefficient  of  variance  (CV),  and  the  percentiles.  The  article  concluded  by 
finding  no  correlation  between  cost  and  schedule  growth.33  In  the  article  “The  Use  of 
Performance  Incentives  in  DoD  Contracting”  (Article  75,  Appendix  A),  the  authors 
employed  empirical  analysis  to  establish  the  relationship  between  the  performance 
achieved  by  contractors  and  certain  variables  such  as  cost  sharing  ratio,  cost,  and 
target  profit. 

E,  STATISTICAL  METHODS 

The  Ch-Squared  Test  provides  a  technique  to  test  the  significance  of  the 
respective  categorical  analysis  across  time  periods.  This  is  achieved  by  calculating 
expected  values  using  the  proportionality  method  in  which  the  number  of  a  certain 
category  (i.e.,  type  of  research  design)  in  a  certain  time  period  is  assumed  to  be 
proportional  to  the  amount  of  the  total  number  of  that  same  category  in  the  total  time  of 
the  publication.  Because  the  Chi-Squared  Tests  become  unstable  when  categories 
contain  few  counts,  three  time  periods  were  used  rather  than  the  standard  four  time 
periods  used  throughout  the  rest  of  the  analysis  in  order  to  achieve  a  larger  sample  size 
per  category,  thus  enabling  the  analysis. 

F.  THEMES 

In  order  to  create  a  consolidated  list  of  major  themes  for  the  ARJ,  both  coders 
took  note  of  what  they  felt  the  theme(s)  were  for  each  of  the  233  articles.  The  themes 
were  extracted  from  the  main  Excel  spreadsheet,  assigned  to  a  single  cell,  and  sorted 
alphabetically  to  determine  recurring  themes.  At  first,  the  list  consisted  of  216  themes, 
but  through  further  discussion  was  eventually  reduced  to  a  list  of  27.  Themes  of  a  similar 
nature  were  grouped  together.  A  list  of  the  theme  categories  and  subcategories  are  listed 
in  Table  5.  Each  one  of  the  233  articles  was  assigned  to  one  of  the  27  theme  categories. 
Eigure  1  in  Chapter  III  shows  the  frequencies  of  the  themes  over  the  13-year  existence  of 
the  ARJ. 


33  Richard  L.  Coleman,  Jessica  R.  Summerville,  and  Megan  E.  Dameron,  “The  Relationship  between 
Cost  Growth  and  Schedule  Growth,”  Acquisition  Review  Journal,  Spring  2003,  p.  120. 
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Themes 

Suhcategories 

Technology 

Anti-tamper,  Information  Technology,  Innovation,  Knowledge 
management,  Net-Centric 

Acquisition  Reform 

Continuous  Improvement,  Restructuring,  Downsizing, 

Transformation 

Contracting 

Auctions,  Buyer-Seller  Relationship,  Contracting  Officer’s 
Technical  Representative  (COTR)  Expert  System  Aid  (CESA), 
Contract  Management,  Support,  Contractor  Performance,  Contract 
Types,  Incentives,  Price-Based  Acquisition,  Request  for  Proposal, 
Service-Level  Agreements,  Sole  Source,  Subcontracting, 
Competition 

Cost,  Schedule,  Perfonnance 

Management,  Estimation,  Activity-Based  Costing,  Budget,  Cost  as 
an  Independent  Variable  (CAIV),  Cycle -time.  Estimate  at 
Completion  (EAC),  Lamed  Value  Management  (EVM),  Total 
Ownership  Cost  (TOC),  Management  Reserve 

Organizational  Behavior 

Cultural  Change,  Human  Resources  Management,  Leadership, 
Organization  Dynamics,  Radical  Change,  Trast,  Workforce 

Education  and  Training 

Defense  Acquisition  Workforce  Improvement  Act  (DAWIA), 
Advanced  Education,  Certification,  Community  of  Practice, 
Instmctional  System  Design 

Logistics 

Electronic  Business,  Depots,  Life-Cycle,  Logistics  Reform,  Supply 
Chain,  Perfonnance-Based  Logistics  (PBL),  Technical  Performance 
Risk  Index  (TPRI) 

Risk 

Management,  Risk  Models,  Metrics,  Cost,  Schedule,  Performance, 
Technology 

Software 

Cyber  warfare.  Development,  Metrics,  Modularity,  Standardization 

Streamlined  Acquisition 

Consolidation,  Restmcturing 

Research,  Development,  Test,  and 
Evaluation 

Technical  Performance  Measures,  Test  and  Evaluation,  Technical 
Evaluation 

Systems  Engineering 

DoD  Architecture  Framework  (DoDAF),  Enterprise  Architecture, 
Integration,  Quality  Assurance,  Quality  Improvement,  Trade-Off 

Laws,  Policies,  and  Regulations 

Buy  America  Act,  Copyright,  National  Security 

Management 

Management  Related  Items 

Change 

Dynamics 

Defense  Industry 

Budget,  Security,  Aircraft,  Commercial  Off  the  Shelf  (COTS), 
Non-Developmental  Items  (NDIs),  Small  Business,  Weapon 
System  Acquisition 

Procurement 

Purchasing 

Decision  Making 

Decision-Making  Process,  Quantitative  Decision  Analysis, 
Strategic  Planning 

International 

Cooperative  Acquisition,  Foreign  Military  Sales,  Globalization, 
Transatlantic 

Program  Management 

Program  Executive  Office/r  (PEO),  Defense  Acquisition  Board 
(DAB) 

System  of  Systems 

Interoperability 

DoD  Framework 

DoD  5000 

Lean  Implementation 

Lean  Six-Sigma 

Industry 

Industrial  and  Commercial  Capability 

Modeling  and  Simulation 

Simulation-Based  Acquisition 

Evolutionary  Acquisition 

Spiral  Development 

Table  5. 

Theme  Categories 

18 


In  the  winter  1997  issue,  after  approximately  four  years  of  publieation, 
Defense  Aequisition  was  defined  as: 

...  the  coneeptualization,  initiation,  design,  development,  logisties 
support,  modification,  and  disposal  of  weapons  and  other  systems, 
supplies,  or  services  to  satisfy  Defense  Department  needs,  or  intended  for 
use  in  military  missions. ^4 

Another  substantial  factor  that  could  have  an  overall  impact  on  the  themes  started 
with  Issue  38,  when  the  ARJ  began  concentrating  on  themed  issues.  Some  of  those 
themes  include  Change,  Risk  Management,  and  Transformation. 

Based  on  the  data  collected,  the  coders  were  able  to  perform  various  analyses 
(which  are  presented  in  Chapter  III),  including  the  results  of  the  comparison  to  Elder’s 
thesis  as  well  as  any  changes  that  occurred  between  2004  and  2006. 


^^Acquisition  Review  Journal,  “Guidelines  for  Authors,”  Winter  1997,  p.  127. 
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III.  RESEARCH  ANALYSIS 


A,  THEMES 

The  first  item  of  interest  is  the  identification  of  themes  throughout  the  existence 
of  the  ARJ.  Analysis  was  performed  to  determine  if  trends  were  formed  based  on 
“hot  issues”  within  the  Acquisition  community.  Part  of  this  analysis  was  to  discover  if 
there  were  any  “hot  button”  issues  discussed  over  time.  How  much  has  been  written 
about  Acquisition  reform?  How  prominent  were  schedule  slips  and  budget  issues  within 
the  publication?  Are  the  themes  very  specific  or  very  broad  across  the  articles  published 
in  the  ARJl  Which  areas  of  the  Acquisition  community  are  under-represented  or 
under-explored? 

The  themes  collected  across  the  years  were  tabulated  to  account  for  their 
frequency  of  usage  and  to  explore  any  patterns.  Although  several  of  the  articles  had 
multiple  themes  and  could  be  assigned  to  more  than  one  theme  category,  it  was  decided 
to  assign  each  article  to  one  theme  category — that  which  best  exemplified  its  main  focus. 
Some  similarities  exist  between  this  study’s  list  of  themes  and  Elder’s  list,  but  not 
unexpectedly,  the  lists  were  not  exact  replicas  of  each  other.  Twenty-seven  categories 
were  agreed  upon  by  the  two  researchers,  while  Elder’s  list  was  comprised  of  15  themes. 
Doubling  the  number  of  bins  available,  this  study  has  more  flexibility  to  spread  out  its 
articles  more  specifically  than  Elder’s  categorical  themes.  In  Appendix  E,  Reform 
Initiatives  was  his  most  common  theme  versus  this  study’s  category  of  Technology.  In 
this  study,  the  theme  of  Cost,  Schedule,  and  Performance  is  ranked  second.  Referring 
back  to  Appendix  E,  when  Cost  and  Schedule  combines  with  Performance  and 
Management,  Elder’s  number  is  very  close  to  what  this  study  arrived  at.  Eooking  for 
more  similarities,  the  number  of  occurrences  for  Elder’s  Reform  Initiatives  theme  is  equal 
to  the  number  of  occurrences  for  the  current  study’s  Acquisition  Reform  and  Streamline 
Acquisition  themes.  Eigure  1  shows  the  number  of  articles  contained  within  each  theme 
category  in  descending  order.  As  a  final  means  of  studying  the  themes  in  the  ARJ,  a 
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sampling  of  articles  is  provided  in  Appendix  B  to  better  illustrate  the  themes  seleeted 
for  categorization. 


Technology 
Cost,  Schedule,  Performance 
Contracting 
Acquisition  Reform 
Education  &  Training 
Risk 
Software 
Logistics 

Laws,  Policies,  and  Regulations 
Organizational  Behavior 
Lean  Implementation 
International 
SE 
RDT&E 
Management 
Defense  Industry 
Change 

Streamline  Acquisition 
Program  Management 
Modeling  and  Simulation 
System  of  Systems 
DoD  Framework 
Decision  Making 
Procurement 
Industry 

Evolutionary  Acquisition 
IPPD/IPT 

0  5  10  15  20  25  30 

Frequency 


Figure  1.  Themes  Categories 

Technology  is  the  most  common  theme  thus  far  in  the  ARJ,  with  Cost,  Schedule, 
and  Performance  close  behind.  Although  Issue  38  started  dedicating  themes  to  eaeh 
issue,  neither  Technology  nor  Cost,  Schedule,  and  Performance  were  one  of  those  issues, 
leading  to  the  belief  that  themed  issues  have  yet  to  impaet  the  trends  presented  thus  far. 
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Starting  in  2005,  all  ARJ  issues  were  assigned  a  speeifie  theme  for  the  authors  to 
eoncentrate  on  in  order  to  be  published.  If  the  ARJ  eontinues  to  publish  issues  based  on 
these  themes,  then,  in  the  long  run,  this  decision  will  have  an  impact  on  the  categorization 
of  themes. 

Elder  broke  his  pool  of  193  articles  into  three  periods  in  order  to  search  for  trends 
over  time.  His  analysis  shows  Management  and  Organizational  Behavior  and 
Interoperability  increased  across  the  time  periods,  while  a  majority  of  the  other 
categories  show  either  a  decrease  or  a  steady  number  of  articles  across  periods.  The 
difference  in  categories  were  apparent;  however,  the  researchers  believed  it  was  based  on 
the  methodology  used  to  derive  their  top  theme  categories,  which  were  primarily  based 
on  interpretation  and  probable  differences  in  their  professional  background  and 
educational  experience. 

Figure  2  shows  the  cumulative  frequency  of  the  occurrences  of  the  233  themes  in 
the  different  time  periods.  Themes  represented  consistently  across  time  periods  include 
Technology,  Cost,  Schedule,  and  Performance',  Contracting',  Acquisition  Reform', 
Education  and  Training',  and  Risk.  Less  than  a  third  of  the  themes  were  localized  to  one 
time  period.  For  example,  although  the  concept  of  lean  implementation  has  been 
important  in  recent  years,  authors  did  not  start  writing  about  it  until  after  2004.  On  the 
other  hand,  the  Federal  Acquisition  Streamlining  Act  was  enacted  in  1994,  and  articles 
about  Streamlining  Acquisition  are  found  exclusively  in  time  period  1,  which  ended  in 
1997.  Laws,  Policies,  and  Regulations  were  written  most  extensively  during  the  first  four 
years  of  the  13-year  time  period.  The  bulk  of  Risk  occurred  in  time  period  3  and 
Logistics  in  time  period  4.  Evolutionary  Acquisition  and  System-of-systems  emerged  as 
topics  during  the  third  period,  suggesting  their  increased  importance  in  defense 
acquisition.  Acquisition  Reform  decreased  as  a  theme  with  each  subsequent  time  period. 
Perhaps  this  is  because  acquisition  reform  is  now  being  tailored  to  specific  areas; 
therefore,  it  may  not  be  prevalent  as  the  main  theme,  but  it  still  exists  within  the 
community.  DoD  Framework  and  Decision  Making  were  topics  of  choice  in  period  1; 
both  of  which  are  making  a  comeback  in  period  4. 
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Figure  2.  Themes  in  Time  Periods 

Figure  2  depicts  a  picture  of  the  themes  per  individual  time  period.  The  most 
frequently-occurring  theme  was  Technology  during  period  3.  In  period  2,  Contracting 
was  the  most  recurring  theme,  thus  ranking  second  across  all  four  time  periods.  Logistics 
consistently  increased  over  the  time  periods,  as  there  was  one  article  in  period  1,  after 
which  the  number  of  articles  doubled  in  period  2,  and  then  tripled  as  the  theme  made  a 
comeback  in  the  fourth  time  period. 
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B. 


CONTRIBUTING  AUTHORS 


The  researchers  felt  it  necessary  to  give  credit  to  the  authors  who  have  contributed 
to  the  ARJ.  First,  the  study  revealed  that  a  total  of  325  authors  have  contributed  articles 
to  the  ARJ.  Based  on  the  number  of  articles  contributed  by  each  author,  90  percent 
contributed  only  one  article.  Table  6  is  a  list  of  top  contributors  for  193  articles,  which 
reflects  three  percent  of  the  authors  and  matches  the  sample  size  in  Elder’s  thesis,  thus 
allowing  direct  comparisons.  These  totals  do  not  reflect  special  issue  introductions  and 
editorial  editions. 


Author(s) 

Contributions 

Christensen,  David  S. 

7 

Washington,  William  N. 

6 

Alford,  Eionel  D. 

4 

Nissen,  Mark  E.,  Dr. 

4 

Graham,  Robert 

3 

Einster,  Bruce  G. 

3 

Pollock,  Neal 

3 

Templin,  Carl 

3 

Table  6.  Top  Contributors  for  193  Articles 

Compared  to  Elder’s  thesis,  there  were  a  few  minor  differences  which  are 
believed  to  be  due  to  data  collection  methodology.  In  the  current  study,  the  two 
researchers  each  maintained  a  separate,  single  spreadsheet.  Each  author  and  co-author 
occupied  an  individual  cell,  allowing  the  researchers  to  perform  a  sort  function  in  Excel, 
which  provided  them  with  an  accurate  count  of  the  authors.  Since  the  researchers 
maintained  their  own  spreadsheets,  the  chance  of  error  was  decreased.  Upon  completion 
of  the  sort  function,  the  results  were  identical.  The  researchers  suspect  Elder  might  have 
used  a  manual  count  method,  leading  to  the  slight  differences  in  totals.  Some  of  the 
leading  authors  were  co-authors  in  a  few  articles,  so  the  chance  of  overlooking  those 
authors  is  highly  possible.  Elder’s  results  can  be  viewed  in  Appendix  D.  Several 
observations  were  as  follows.  Dr.  David  Christensen  was  the  top  contributor  according 
to  both  studies.  However,  notice  that  this  study  counted  seven  articles  for  this  author, 
while  Elder  found  six  articles  in  which  included  an  issue  introduction. 
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Dr.  William  Washington  contributed  six  articles  according  to  this  study;  but  according  to 
Elder’s,  he  contributed  five  articles.  On  the  other  hand,  this  study  showed  four  article 
contributions  for  Dr.  Mark  Nissen,  while  Elder  noted  five  contributions.  The 
contributions  of  Et  Col  Eionel  Alford  (four  articles),  Lt  Col  Bruce  Einster, 
Mr.  Neal  Pollock,  and  Dr.  Carl  Templin  (three  articles)  agreed  in  both  studies. 
Mr.  Robert  Graham  was  included  among  the  top  contributors  because  of  his  submission 
of  three  articles.  Elder’s  list  of  top  contributors  included  the  following  authors; 
Maj  Joseph  Besselman,  Et  Col  Driessnack,  Dr.  Keith  Snider,  Mr.  Ashish  Arora,  and 
Mr.  Patrick  Earkey;  however,  they  are  not  reflected  in  the  current  list.  All  authors 
mentioned  were  noted  with  three  articles  in  Elder’s  study,  but  only  two  articles  were 
found  for  each  author  in  this  study. 

The  number  of  authors  that  contributed  to  the  ARJ  did  not  change  substantially 
from  2004  to  2006.  Table  7  represents  the  Top  Contributors  for  233  articles. 
Daniel  J.  Sherman  joined  Mark  E.  Nissen  and  Eionel  D.  Alford  in  the  top  four  authors, 
but  otherwise  the  list  was  unchanged  from  Table  6. 


Author(s) 

Contributions 

Christensen,  David  S. 

7 

Washington,  William  N. 

6 

Alford,  Eionel  D. 

4 

Nissen,  Mark  E.,  Dr. 

4 

Sherman,  Daniel  J. 

4 

Graham,  Robert 

3 

Einster,  Bruce  G. 

3 

Pollock,  Neal 

3 

Templin,  Carl 

3 

Table  7.  Top  Contributors  for  233  Articles 


Table  8  shows  the  number  of  authors  per  article  per  time  period.  An  overall 
picture  shows  that  roughly  60  percent  of  the  articles  were  written  by  a  single  author. 
Twenty-two  percent  of  the  articles  were  tendered  by  two  authors,  15  percent  had 
three  authors,  and  the  remaining  had  four  or  more  authors  counting  one  article  out  of  233 
that  had  five  authors.  The  mean  (average  number)  of  the  authors  per  article  was 
calculated  based  on  the  number  of  contributing  authors  and  the  number  of  articles 
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published  each  year.  For  example,  in  1994,  21  authors  contributed  to  the  ARJ  and 
18  articles  were  published;  therefore,  there  were  1.17  authors  per  article.  Prior  to  2001, 
the  number  of  articles  written  by  one  author  remained  steady.  In  2002,  it  dropped  by 
more  than  half  and  there  was  an  increase  in  the  number  of  two  or  more  authors  per 
article.  Elder’s  results  up  until  2004  were  relatively  similar  to  these  results. 


Year 

No.  of 
Authors 

No.  of 
Articles 

Mean 

One 

Author 

Two 

Authors 

Three 

Authors 

Four  + 
Authors 

1994 

21 

18 

1.17 

83.3% 

16.7% 

0.0% 

0.0% 

1995 

14 

10 

1.40 

70.0% 

20.0% 

10.0% 

0.0% 

1996 

13 

11 

1.18 

81.8% 

18.2% 

0.0% 

0.0% 

1997 

44 

25 

1.76 

60.0% 

8.0% 

28.0% 

4.0% 

1998 

32 

20 

1.60 

60.0% 

25.0% 

10.0% 

5.0% 

1999 

38 

24 

1.58 

62.5% 

20.8% 

12.5% 

4.2% 

2000 

39 

22 

1.77 

63.6% 

13.6% 

9.1% 

13.6% 

2001 

18 

14 

1.29 

78.6% 

14.3% 

7.1% 

0.0% 

2002 

42 

19 

2.21 

31.6% 

26.3% 

31.6% 

10.5% 

2003 

30 

17 

1.76 

52.9% 

17.6% 

29.4% 

0.0% 

2004 

27 

13 

2.08 

23.1% 

46.2% 

30.8% 

0.0% 

2005 

50 

28 

1.79 

46.4% 

35.7% 

10.7% 

7.1% 

2006 

17 

12 

1.42 

58.3% 

41.7% 

0.0% 

0.0% 

Table  8.  Authors  per  Article  per  Time  Period 

As  found  in  Carter  and  Ehram’s  article,  the  “significantly  greater  number  of 
authors  per  article  over  time  is  similar  to  the  pattern  that  has  been  documented  in  other 
academic  discipline. According  to  them,  researchers’  thought  process  is  the  level  of 
productivity  is  greater  when  the  responsibility  is  shared  by  fellow  researchers. ^6 

An  analysis  of  the  number  of  contributing  authors  per  article  during  the  ARJ’s 
13-year  existence  followed  using  ANOVA  and  the  Chi-Squared  Test.  The  researchers 
maintained  Elder’s  breakdown  of  four  time  periods.  Eor  both  techniques,  the 
hypotheses  were: 

Ho:  No  difference  exists  in  the  number  of  authors  per  article  across  time  periods. 

Hi:  A  difference  exists  in  the  number  of  authors  per  article  across  time  periods. 


35  Craig  R.  Carter  and  Lisa  M.  Ellram,  “Thirty-Five  Years  of  the  Journal  of  Supply  Chain 
Management:  Where  Have  We  Been  and  Where  are  We  Going?,”  Journal  of  Supply  Chain  Management, 
Spring  2003,  pp.  37-38. 

36  Ibid.,  p.  38. 
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For  the  ANOVA  technique,  the  dependent  variable  was  the  number  of  authors  per 
article,  while  the  independent  variable  was  the  time  periods  ranging  from  time  period 
1  =  1994-1997,  period  2  =  1998-2000,  period  3  =  2001-2003,  and  period  4  =  2004-2006. 
Appendix  H  supports  the  null  hypothesis;  there  is  not  enough  evidence  to  say  differences 
exist  in  the  number  of  authors  per  article  across  time  periods  (p  =  0.1115  and 
F  =  2.0228). 

On  the  other  hand,  the  Chi-Squared  Test  revealed  a  different  result.  For  the 
expected  frequencies,  the  researchers  made  an  assumption  that  the  number  of 
one-authored  articles  in  a  particular  time  period  is  proportional  to  the  number  of 
one-authored  articles  in  the  total  number  of  articles  for  all  time  periods.  For  example, 
64  one-authored  articles  out  of  233  total  articles  yield  27.5  percent.  If  this  assumption 
were  true,  then  in  time  period  1,  27.5  percent  of  the  136  articles  (total  for  time  period  1) 
yields  37  expected  one-author  articles.  With  a  p-value  of  0.0099,  the  researchers  have 
enough  evidence  to  reject  the  null  hypothesis,  meaning  there  is  a  difference  in  the  number 
of  authors  across  time  periods  (Appendix  K). 

Again,  this  study  is  interested  in  whether  its  findings  agreed  with  Elder’s  findings. 
Elder’s  study  was  replicated  by  dividing  the  present  data  into  identical  time  periods  in 
line  with  his  time  period  divisions,  which  consisted  of  a  four-,  three-,  and  four-year 
breakdown.  Time  periods  were  period  1  =  1994-1997,  period  2  =  1998-2000,  and  period 
3  =  2001-2004. 

Referring  to  Appendix  E,  for  the  identical  time  period,  the  ANOVA  analysis 
exposed  a  difference  in  the  number  of  authors  per  article  per  time  period  (p  =  0.0341). 
While  Elder’s  p-value  was  0.0205,  he  also  had  the  same  findings-  that  the  results  suggest 
a  statistically  significant  overall  difference  across  time  periods.  The  difference  in 
p-values  probably  relates  to  a  minor  counting  difference  in  the  number  of  authors;  recall 
that  this  study’s  count  of  leading  authors  differed  from  Elder’s  count  and  those 
differences  might  reflect  a  common  cause  of  slightly  different  counts  for  numbers  of 
authors.  Both  findings  discovered  a  similar  upward  trend,  as  did  Carter  and  Ellram’s  in 
their  study  of  the  Journal  of  Supply  Chain  Management,  where  the  mean  of  authors  per 
article  increased  over  time. 
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Furthermore,  we  examined  the  difference  between  the  results  of  the  ANOVA  for 
the  articles  through  part  of  year  2004  (Elder’s  sample)  and  the  articles  until  year  2006. 
Referring  back  to  Table  8,  the  means  were  elevated:  of  2.21  for  2002  and  2.08  for  2004. 
These  two  means  joined  together  in  the  same  time  period  (2001-2004),  as  replicated  from 
Elder’s  study)  elevated  the  mean  for  the  second  (through  2004)  ANOVA  analysis.  In 
part  one  (until  2006),  where  233  articles  were  counted,  the  two  means  were  spread  out  in 
two  time  periods  of  three  and  four.  Therefore,  the  variability  was  spread  out  thinner 
across  six  years. 

Eigure  3  shows  the  number  of  articles  published  per  year  (year  2006  is  not  a 
complete  year).  It  does  not  show  a  noticeable  trend  of  increasing  number  of  articles  over 
time,  although  there  were  significant  spikes  from  1996  to  1997  and  2004  to  2005.  There 
does  not  appear  to  be  any  relationship  between  the  maturation  of  the  ARJ  and  the  number 
of  articles  being  published. 


Eigure  3 .  Number  of  Articles  per  Y ear 

1,  Author’s  Institutional  Affiliation 

ARJ  solicits  from  any  prospective  authors  who  have  published  and  have  expertise 
in  the  subject  and  who  are  involved  in  the  defense  acquisition  process.  Table  9  shows  the 
organizations  included  under  each  of  the  coder’s  final  categories. 
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Institutional  Category 

Inclusion 

Civilian  Universities 

All  civilian  academic  universities  and  colleges,  both 
graduate  and  post  graduate  levels 

Civilian  Organizations 

Non-DoD  and  non-services  organizations 

Military  Universities 

Includes  AFIT,  NPS,  USAF  Academy,  Naval  Academy, 
Army  Academy,  and  U.S.  Military  Academy 

Other  USA 

Non-academic  U.S.  Army  facilities  and  organizations 
i.e..  Army  Research  Laboratory’s  Aerodynamics 

Branch,  U.S.  Army  Corps  of  Engineer,  and  1-36 

Infantry 

Other  USAF 

Non-academic  U.S.  Air  Force  facilities  and 
organizations,  i.e.,  F/A  22  Systems  Program  Office 

SPO,  Joint  Strike  Fighter  JSF  SPO,  and  Special 
Operations  Command 

OSD 

Office  of  Under  Secretary  of  Defense,  Acquisition, 
Technology,  and  Logistics  (USD  (AT&L)),  Weapons 
Systems  Cost  Analysis  Division,  and  Management 
Policy  and  Program  Integration  Division 

ACSC/DSMC 

Faculty  and  students  of  Air  Command  Staff  College  and 
Defense  Systems  Management  College 

Other  DoD 

Includes  Ballistic  Missile  Defense  Organization 

DAU 

Faculty  and  students  of  DAU 

AWC/ICAF/NWC 

Faculty  and  students  of  Air  War  College,  Industrial 
College  of  the  Armed  Forces,  and  Naval  War  College 

Other  USN 

Non-academic  U.S.  Navy  facilities,  i.e..  Naval  Center 
for  Cost  Analysis,  Department  of  the  Navy,  Chief 
Information  Office,  and  Office  of  Naval  Operations 

Other  GOV 

General  Accounting  Office,  National  Aeronautics  Space 
Administration  (NASA),  and  Atlantic  Council  of  the 
United  States 

Other  USMC 

Non-academic  U.S.  Marine  Corps  facilities,  i.e..  Marine 
Corps  Systems  Command,  Quantico,  VA  and  Depot 
Maintenance  Business  Area,  HQ  Marine  Corps 

Civilian  Research 

Institutions 

Includes  MITRE  Corporation,  International  Research 
Institute,  and  Nichols  Research  Eab 

Table  9.  Institutional  Categories’  Inclusions 

Figure  4  shows  that  most  contributors  come  from  civilian  universities,  followed 
by  civilian  organizations.  Military  universities  such  as  the  Air  Force  Institute  of 
Technology  (AFIT),  Naval  Postgraduate  School  (NPS),  Air  Force  Academy,  and  the 
Naval  Academy  made  up  13  percent.  Contributions  from  civilian  research  institutions 
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such  as  MITRE  Corporation,  Nichols  Research  Lab,  and  International  Researeh  Institute 
made  up  less  than  two  percent.  The  United  States  Navy  (USN)  and  the  United  States 
Marine  Corps  (USMC)  eontributed  approximately  two  percent,  six  to  ten  pereent  lower 
than  the  United  States  Air  Foree  (USAF)  and  United  States  Army  (USA). 


Institutional  Categories 


0.0%  2.0%  4.0%  6.0%  8.0%  10.0%  12.0%  14.0%  16.0%  18.0%  20.0% 
Percent  Frequency 


Figure  4.  Author’s  Institutional  Affiliation 

In  Table  10,  the  researehers  wanted  to  diseover  if  there  were  evident  trends  based 
on  the  institutional  affiliation  of  the  contributing  authors.  (Note:  Authors  who  have 
revealed  their  affiliation.) 
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Institutional  Categories 

Period  1 
1994-1997 

Period  2 
1998-2000 

Period  3 
2001-2003 

Period  4 
2004-2006 

Civilian  Universities 

10 

25 

12 

17 

Other  USA 

14 

7 

14 

7 

Military  Universities 

18 

12 

12 

5 

Civilian  Organizations 

10 

14 

3 

11 

Other  USAF 

2 

16 

4 

10 

Office  of  Secretary  of  Defense  (OSD) 

7 

5 

3 

7 

Air  Command  Staff  College/Defense 
Systems  Management  College 
(ACSC/DSMC) 

12 

8 

2 

1 

Other  DoD 

3 

2 

2 

2 

DAU 

2 

0 

0 

10 

Air  War  College/Industrial  College  of 
the  Air  Force/  Naval  War  College 
(AWC/ICAF/NWC) 

4 

0 

1 

1 

Other  USN 

2 

3 

2 

0 

Other  Government  (GOV) 

1 

1 

1 

1 

Other  USMC 

1 

2 

0 

3 

Civilian  Research  Institutions 

3 

1 

0 

2 

Table  10.  Institutional  Categories  Divided  into  Time  Periods 


The  DAU  had  an  inerease  in  contributions  during  the  fourth  period.  Military 
communities  started  off  strong  in  the  first  three  periods  and  then  experienced  a  dramatic 
decrease,  whereas  the  Civilian  Universities  were  very  dedicated  contributors  over  all  four 
periods.  Catering  to  the  higher  level  of  the  Acquisition  Corps,  it  is  ironic  that  DoD  did 
not  contribute  many  articles.  Elder’s  findings  display  the  same  results  and  there  are  no 
significant  discrepancies  (see  Appendix  G). 

2,  Author’s  Educational  Affiliation 

Out  of  the  total  number  of  authors,  243  revealed  their  highest  degree  of  education; 
122  authors  have  doctorates,  110  have  master’s  degrees,  and  11  have  bachelor’s  degrees. 
In  their  analysis,  the  researchers  also  noted  those  authors  with  multiple  degrees. 
Twenty-six  authors  have  more  than  one  master’s  degree,  which  is  roughly  10.7  percent  of 
the  authors  who  noted  their  degrees. 
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Table  11  lists  the  Universities  and  Colleges  of  those  authors  who  noted  their 
highest  aeademie  degrees.  Industrial  College  of  the  Armed  Forees  (ICAF)  graduates 
have  the  largest  eontribution,  followed  by  graduates  of  the  Air  Foree  Institute  of 
Teehnology  (AFIT)  and  the  Naval  Postgraduate  Sehool  (NPS). 


University/College/Institutions 

Contribution 

Industrial  College  of  the  Armed  Forces 

14 

Air  Force  Institute  of  Technology 

12 

Naval  Postgraduate  School 

12 

University  of  California* 

11 

University  of  Southern  California 

11 

George  Washington  University 

9 

Boston  University 

9 

George  Mason  University 

8 

Defense  Systems  Management  College 

7 

University  of  Alabama 

7 

Stanford  University 

6 

Texas  A&M  University 

6 

University  of  Nebraska 

6 

Georgia  Institute  of  Technology 

5 

Rennselaer  Polytechnic  Institute 

5 

University  of  Michigan 

5 

Arizona  State  University 

4 

Carnegie  Mellon  University 

4 

Massachusetts  Institute  of  Technology 

4 

University  of  Maryland 

4 

University  of  Virginia 

4 

*Berkeley,  Irvine,  and  Los  Angeles  eampuses 


Table  1 1 .  Author’s  Edueational  Affiliation 

The  eivilian  edueational  institutions  appear  to  be  the  front-runners  for  published 
artieles  in  the  ARJ,  although  the  top  three  are  military  institutions. 

3,  Academe  or  Practitioner 

The  ARJ  welcomes  authors  from  academe  and  those  in  practice.  The  authors 
were  classified  into  categories  that  identify  them  as  academe  or  practitioners  within  the 
defense  acquisition  industry.  Consistent  with  the  other  elements  of  this  study,  the  time 
periods  were  divided  into  three  time  periods  and  four  time  periods.  Table  12  ends  in 
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period  3,  replicating  Elder’s  thesis.  His  results  are  located  in  Appendix  H.  Table  13 
shows  the  four  time  periods  that  extend  to  233  articles. 


Type 

Period  1 
1994-1997 

Period  2 
1998-1999 

Period  3 
2000-2004 

Academe 

44.8% 

45.8% 

43.0% 

Practitioner 

55.2% 

54.2% 

57.0% 

Table  12.  Proportion  of  Academe  and  Practitioner  Contributions  (193  Articles) 

On  average,  55  percent  of  the  contributors  were  practitioners.  The  results  of  this 
study  were  in  line  with  Elder,  who  obtained  a  finding  of  58  percent. 


Type 

Period  1 
1994-1997 

Period  2 
1998-1999 

Period  3 
2000-2003 

Period  4 
2004-2006 

Academe 

44.8% 

45.8% 

45.1% 

38.1% 

Practitioner 

55.2% 

54.2% 

54.9% 

61.9% 

Table  13.  Proportion  of  Academe  and  Practitioner  Contributions  (233  Articles) 

Table  13  shows  a  predominance  of  practitioners  over  academe  contributors.  The 
increase  in  the  magnitude  of  the  difference  in  period  4  is  inconclusive  because  2006  is 
not  representative  of  a  full  year’s  publication. 

4,  Civilian  or  Government  Contributions 

Another  investigative  question  this  study  attempted  to  answer  is  the  frequency  of 
civilian  and  government  contributions.  The  ARJ  is  published  by  a  defense  university; 
however,  it  solicits  articles  from  anyone  interested  in  defense  acquisition.  Table  14 
represents  the  proportion  of  civilian  and  government  contributors  based  on  193  articles. 


Type 

Period  1 
1994-1997 

Period  2 
1998-2000 

Period  3 
2001-2004 

Civilian 

25.0% 

41.1% 

49.5% 

Government 

75.0% 

58.9% 

50.5% 

Table  14.  Proportion  of  Civilian  and  Government  Contributions  (193  Articles) 

The  proportion  of  civilian  and  government  contributions  aligned  with  Elder’s 
results  found  in  Appendix  I.  Government  contributed  three  times  the  amount  of  civilian 

34 


contributions  in  the  first  four  years  of  publication,  but  it  appeared  to  diminish  in  the  next 
time  periods.  Civilian  eontributions  started  to  ramp  up  as  the  years  went  on.  Table  15 
shows  that  the  additional  time  period  for  the  overall  study  eontinued  to  exhibit  decreasing 
contributions  from  the  government. 


Type 

Period  1 
1994-1997 

Period  2 
1998-2000 

Period  3 
2001-2003 

Period  4 
2004-2006 

Civilian 

25.0% 

41.1% 

43.7% 

44.2% 

Government 

75.0% 

58.9% 

56.3% 

55.8% 

Table  15.  Proportion  of  Civilian  and  Government  Contributions  (233  Articles) 

C.  CATEGORICAL  ANALYSIS 

Categorieal  Analysis  consists  of  three  parts.  First,  the  researehers  want  to 
replicate  Elder’s  thesis.  After  that,  an  independent  analysis  will  be  performed  including 
the  artieles  that  have  been  published  from  2004  to  2006.  Finally,  a  statistieal  analysis 
will  show  the  changes  and  trends  diseovered. 

1,  Comparison  with  Elder’s  Thesis  (2005) 

The  researchers  divided  the  years  in  aeeordanee  to  Elder’s  time  periods  in  order  to 
replicate  his  study.  Table  16  illustrates  the  results  of  the  type  of  researeh  performed. 
Elder’s  results  are  found  in  Appendix  M. 


Type 

Period  1 

Period  2 

Period  3 

1994-1997 

1998-2000 

2001-2004 

Normative  Review 

25.0% 

28.8% 

25.4% 

Eiterature  Review 

20.3% 

27.3% 

14.3% 

Exploratory  Studies 

28.1% 

24.2% 

23.8% 

Methodology  Review 

20.3% 

10.6% 

19.0% 

Hypothesis  Testing 

6.3% 

9.1% 

17.5% 

Table  16.  Types  of  Researeh  Performed 


a.  Type  of  Research  Performed 

Although  the  numbers  between  the  two  studies  are  not  identieal  due  to 


differences  in  the  interpretation  of  the  definitions  (the  edueational  and  professional 
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background  of  the  researchers  perhaps  contributed  to  the  disparity)  and  individual  bias,  it 
did  not  prevent  the  noticing  of  similarities  between  the  two.  One  similarity  is  that  the 
research  published  in  ARJ  is  primarily  qualitative  rather  than  quantitative  research.  Both 
studies  show  that  roughly  45  percent  of  the  type  of  research  performed  has  been  either 
Normative  or  Literature  reviews,  and  another  48  percent  has  been  Exploratory  Research 
or  Methodology  reviews  (“how-to”  papers).  Another  similarity  is  that  Hypothesis 
Testing  has  been  the  least  common  type  of  research  performed.  It  is  believed  that  there 
has  been  less  quantitative  research  noted  in  the  ARJ  because  over  time  there  have  been 
many  themes  represented  in  the  journal;  therefore,  it  has  not  been  possible  to  reach  a 
point  where  researchers  can  begin  quantitative  analysis.  In  other  words,  there  is 
insufficient  concentration  on  a  specific  area  long  enough  to  get  enough  data  to  conduct 
solid  quantitative  analysis. 

b.  Type  of  Research  Design 

After  categorizing  the  types  of  research  performed,  the  types  of  research 
designs  are  reflected  in  Table  17. 


Type 

Period  1 

Period  2 

Period  3 

1994-1997 

1998-2000 

2001-2004 

Survey 

1.6% 

13.6% 

6.3% 

Case  Study 

50.0% 

36.4% 

42.9% 

Interviews 

9.4% 

7.6% 

14.3% 

Archival  Study 

3.1% 

12.1% 

12.7% 

Experiment 

0.0% 

4.5% 

4.8% 

Simulation 

1.6% 

1.5% 

0.0% 

Mathematical  Model 

4.7% 

4.5% 

1.6% 

Topic  Presentation 

29.7% 

19.7% 

17.5% 

Table  17.  Types  of  Research  Design 

The  results  for  the  type  of  research  design  aligned  with  the  results  of  the 
types  of  research  performed.  Both  studies  reported  that  qualitative  analysis  (Case  Studies 
and  Topic  Presentation)  made  up  substantially  more  than  50  percent  of  the  articles  that 
were  published.  These  design  types  are  used  to  inform  about  and  introduce  topics  that 
need  further  research.  It  is  interesting  that  both  Case  Studies  and  Topic  Presentations 
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have  been  on  a  steady  decline,  offset  by  increases  in  interviews,  but  also  in  Surveys  and 
Archival  Studies,  both  of  which  can  be  more  quantitative;  a  further  increase  in  the 
quantitative  areas  could  be  apparent  in  the  next  few  years.  Elder’s  results  can  be  viewed 
in  Appendix  M. 


c.  Type  of  Data  Analysis 


Type  of  Data  Analysis  is  the  third  category  of  the  content  analysis  and  the 
results  of  the  current  study  are  shown  in  Table  18. 


Type 

Period  1 

Period  2 

Period  3 

1994-1997 

1998-2000 

2001-2004 

Anecdotal  Evidence 

7.8% 

6.1% 

14.3% 

Statistical  Analysis 

1.6% 

15.2% 

11.1% 

Content  Analysis 

29.7% 

28.8% 

49.2% 

Comparison  Analysis 

48.4% 

36.4% 

12.7% 

No  Analysis 

12.5% 

13.6% 

12.7% 

Table  18.  Types  of  Data  Analysis 

This  study’s  results  aligned  with  the  results  of  the  other  two  categories. 
Statistical  Analysis  did  show  an  increase  after  1998  (although  a  downturn  in  period  3 
followed  the  upturn  in  period  2).  The  qualitative  areas  (Anecdotal  Evidence,  Content 
Analysis,  Comparison  Analysis,  and  No  Analysis)  dominate  the  ^i?Jthus  far,  as  shown  in 
both  studies.  Elder’s  results  can  be  viewed  in  Appendix  M. 

In  all  three  categorical  areas,  the  trends  are  similar  in  both  studies,  but  the 
numbers  are  not  distributed  evenly  and  this  is  conceivably  due  to  differences  in  methods 
and  interpretations.  The  two  studies  could  have  conducted  their  data  collection 
differently  and  their  process  of  settling  any  discrepancies  could  affect  the  overall 
outcome  of  the  data.  A  possible  methodological  difference  may  be  the  manner  in  which 
the  articles  were  read  and  categorized.  The  text  of  each  article  was  reviewed  by  the  two 
researchers  that  resulted  in  two  different  sets  of  results. 

2,  Aggregate  Analysis  of  233  Articles  until  2006 

This  section  reflects  the  results  of  233  articles  that  have  been  published  in  the 

ARJ.  As  the  publication  went  on  beyond  Elder’s  initial  study,  the  researchers  want  to 
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determine  if  the  trends  remained  or  shifted  in  a  different  direction.  In  this  section,  due  to 
the  increase  in  the  number  of  articles  (233  articles),  dividing  into  four  time  periods  is 
more  appropriate. 


a.  Type  of  Research  Performed 

Figure  5  shows  the  trends  of  the  type  of  research  performed  based  on  233 
articles.  Throughout  the  periods  Normative  review  has  dominated.  The  number  of 
articles  involved  in  Normative  Review  combined  with  those  representing  Literature 
reviews  suggests  that  the  articles  published  in  the  ARJhsLVQ  a  “large  degree  of  substantive 
justification  but  little  subsequent  theory  development  and  testing. ”37  Although 
Hypothesis  Testing  has  not  been  very  popular  over  the  periods,  there  was  a  definite 
increase  in  the  third  period,38  followed  by  a  dramatic  decrease  in  the  fourth  period. 


Figure  5.  Type  of  Research  Performed 


37  John  Mentzer  and  Kenneth  Kahn,  “A  Framework  of  Logistics  Research,”  Journal  of  Business 
Logistics,  Vol.  16,  No.  1,  1995,  p.  241. 

3^  Consistent  with  Carter  and  Ellram’s  findings  that  the  “use  of  hypothesis  did  indeed  show  a 
statistically  significant  increase  over  time.” 
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b.  Type  of  Research  Design 


In  Figure  6,  Case  Studies  are  shown  as  the  dominant  research  design 
across  all  four  periods.  The  publishes  articles  that  cover  a  wide  array  of  topics  from 
everyday  issues  to  unique  programs  that  survived  the  Acquisition  Process.  Archival 
Studies  are  the  second  most  prominent  research  design  in  the  ARJ.  This  emphasis  on 
Case  Studies  especially  aligns  with  the  trend  favoring  qualitative  data.  Archival  Studies 
are  very  useful  in  the  Acquisition  community  because  many  practitioners  learn  best 
through  past  experiences.  Since  the  ARJ  described  as  a  forum  to  exchange  opinions, 
communicate  policy  decisions,  and  maintain  awareness  for  acquisition  management 
philosophies,  these  trends  are  aligned  with  its  mission.  Consistent  with  both  Carter  and 
Ellram  and  Elder’s  findings.  Experiments,  Simulations,  and  Mathematical  Modeling  were 
less  utilized.39 
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Eigure  6.  Type  of  Research  Design 


39  Craig  Carter  and  Lisa  Ellram,  “Thirty  Five  Years  of  the  Journal  of  Supply  Chain  Management: 
Where  Have  We  Been  and  Where  are  We  Going?,”  Journal  of  Supply  Chain  Management,  Spring  2003, 
p.  32. 
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c. 


Type  Data  Analysis 


Figure  7  shows  a  marked  deeline  in  the  use  of  Comparison  Analysis,  but 
Content  Analysis  has  been  eonsistent  throughout  the  periods.  Statistieal  Analysis  is  the 
least  eommon  throughout  the  periods;  however,  it  has  inereased  since  the  first  period. 
Although  the  research  in  the  ARJ  has  been  more  qualitative  that  quantitative,  some 
evidence  supports  the  argument  that  research  published  in  the  ARJ  is  moving  in  a 
quantitative  direction. 


Data  Analysis 
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1994-1997 


Period  2 
1998-2000 


Period  3 
2001-2003 


Period  4 
2004-2006 


Time  Period 


□  Anecdotal  Evidence 

□  Statistical  Analysis 

□  Content  Analysis 

□  Comparison  Analysis 
■  No  Analysis 


Figure  7.  Data  Analysis  Method 

It  is  worth  noting  that  using  quantitative  data  does  not  automatically  imply 
a  quantitative  data  analysis.^*’ 

3,  Statistical  Analysis  of  the  Categories 

Throughout  this  study,  the  233  articles  have  been  divided  into  four  time  periods 
for  the  aggregate  study  and  divided  into  three  time  periods  to  do  a  fair  comparison  with 
Elder’s  thesis.  In  order  to  sufficiently  aggregate  the  quantities  within  categories  to  permit 
categorical  analysis,  in  this  part  of  the  research,  the  articles  were  divided  into  three  time 


Janet  A.  Harkness,  Fons  .J.  R.  van  de  Vijver,  and  Peter  Mohler,  Cross-Cultural  Survey  Methods, 
Wiley,  Hoboken,  NJ,  2003,  p.  36. 
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periods.  This  is  to  achieve  a  better  sample  size  (greater  than  five  for  each  cell)  in  the 
category  tables  to  perform  a  valid  Chi-Squared  Test.  The  Chi-Squared  Test  was 
performed  to  find  differences,  if  any,  in  the  number  of  respective  types  based  on  three 
separate  categorical  periods.  The  first  time  period  consists  of  five  years  (1994-1998),  the 
second  time  period  consists  of  four  years  (1999-2002),  and  the  third  time  period  also 
consists  of  four  years  (2003-2006). 

For  all  categories,  this  study  assumes  that  there  are  no  differences  in  the  types  of 
articles,  types  of  research  categories,  and  types  of  data  analysis  employed  across  time 
periods.  The  respective  hypotheses  are  stated  in  each  category. 

a.  Type  of  Research  Performed 

Tables  19  and  20  give  the  actual  and  expected  frequencies  for  the  type  of 
research  performed.  The  values  in  these  tables  were  used  to  calculate  the 
Chi-Squared  Test. 

Ho:  No  difference  in  the  type  of  article  category  across  time  periods. 

Hi:  A  difference  in  the  type  of  article  category  exists  across  time  periods. 


Type 

Period  1 

Period  2 

Period  3 

1994-1998 

1999-2002 

2003-2006 

Normative 

22 

21 

23 

Literature  Review 

19 

15 

11 

Exploratory  Studies 

23 

21 

18 

Methodology  Review 

15 

9 

13 

Hypothesis  Testing 

5 

22 

5 

Table  19.  Actual  Frequency  for  Type  of  Research  Performed 


Type 

Period  1 

Period  2 

Period  3 

1994-1998 

1999-2002 

2003-2006 

Normative 

23 

24 

19 

Literature  Review 

16 

16 

13 

Exploratory  Studies 

22 

23 

18 

Methodology  Review 

13 

13 

11 

Hypothesis  Testing 

11 

12 

9 

Table  20.  Expected  Frequency  for  Type  of  Research  Performed 
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Employing  the  same  assumption,  the  expeeted  frequeneies  eonsistently 
use  a  proportionality  method.  When  the  null  hypothesis  is  true,  the  observed  results  and 
the  expeeted  results  should  be  similar.  On  the  other  hand,  if  the  null  hypothesis  is  not 
true,  some  of  the  observed  and  expeeted  results  will  be  different.  The  Chi-Squared  Test 
resulted  in  a  p-value  of  0.0126.  Therefore,  the  researchers  can  reject  the  null  hypothesis; 
there  is  a  difference  in  the  type  of  article  categories  across  time  periods. 

b.  Type  of  Research  Design 

Tables  21  and  22  give  the  actual  and  expected  frequency  for  the  type  of 
research  design.  The  values  in  these  tables  were  used  to  calculate  the  Chi-Squared  Test. 

Ho:  No  difference  in  the  type  of  research  designs  across  time  periods. 

Hi:  A  difference  in  the  type  of  research  designs  exists  across  time  periods. 


Type 

Period  1 
1994-1998 

Period  2 
1999-2002 

Period  3 
2003-2006 

Survey  and  Interviews 

12 

17 

11 

Case  Study/ies 

40 

28 

25 

Archival  Study,  Experiment, 
Simulation,  and  Mathematical  Model 

9 

17 

15 

Topic  Presentation 

23 

17 

19 

Table  21.  Actual  Frequency  for  Type  of  Research  Design 


Type 

Period  1 
1994-1998 

Period  2 
1999-2002 

Period  3 
2003-2006 

Survey  and  Interviews 

14 

14 

12 

Case  Study/ies 

34 

32 

28 

Archival  Study,  Experiment, 
Simulation,  and  Mathematical  Model 

15 

14 

12 

Topic  Presentation 

21 

20 

18 

Table  22.  Expected  Frequency  for  Type  of  Research  Design 


The  p-value  for  this  Chi-Squared  Test  is  0.2739,  which  is  greater  than 
0.05.  The  researchers  combined  Archival  Study,  Experiment,  Simulation,  and 
Mathematical  Modeling  because  the  four  research  designs  share  a  quantitative  nature. 
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Due  to  the  similarities,  they  ean  be  eombined  to  produee  a  more  valid  test  result. 
Regardless,  the  Chi-Squared  Test  eoneludes  that  there  is  no  differenee  in  the  type  of 
researeh  eategories  aeross  time  periods. 

c.  Type  of  Data  Analysis 

Tables  23  and  24  give  the  aetual  and  expeeted  frequency  for  the  type  of 
data  analysis.  The  values  in  these  tables  were  used  to  calculate  the  Chi-Squared  Test. 

Hq:  No  difference  in  the  type  of  data  analysis  across  time  periods. 

Hi:  A  difference  in  the  type  of  data  analysis  exists  across  time  periods. 


Type 

Period  1 

Period  2 

Period  3 

1994-1998 

1999-2002 

2003-2006 

Anecdotal  Evidence 

7 

6 

12 

Statistical  Analysis 

3 

11 

8 

Content  Analysis 

23 

32 

27 

Comparison  Analysis 

38 

22 

8 

No  Analysis 

13 

8 

15 

Table  23.  Actual  Frequency  for  Type  of  Data  Analysis 


Type 

Period  1 

Period  2 

Period  3 

1994-1998 

1999-2002 

2003-2006 

Anecdotal  Evidence 

9 

8 

8 

Statistical  Analysis 

8 

7 

7 

Content  Analysis 

30 

28 

25 

Comparison  Analysis 

25 

23 

20 

No  Analysis 

13 

12 

11 

Table  24.  Expected  Frequency  for  Type  of  Data  Analysis 


The  Chi-Squared  Test  showed  a  p-value  of  0.0003.  In  period  1, 
Statistical  Analysis  was  used  in  only  three  studies.^i  Although  some  statisticians  prefer 
more  than  five  frequencies  in  each  cell,  the  required  condition  specifically  applies  to  the 
expected  frequencies.  The  test  performed  in  this  section  still  fulfilled  the  required 


The  actual  sampling  distribution  of  the  test  statistic  is  discrete  but  it  can  be  approximated  by  the 
chi-squared  distribution  provided  that  the  sample  size  is  large.  The  sample  size  must  be  large  enough  so 
that  the  expected  value  for  each  cell  must  be  five  or  more.  Gerald  Keller,  Statistics  for  Management  and 
Economics,  7th  Edition,  Thomson/Brooks/Cole,  2005,  pp.  554-555. 
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condition,  despite  the  low  actual  count  of  Statistical  Analysis  in  Period  1 .  Therefore,  a 
conelusion  ean  be  drawn  from  the  analysis  that,  based  on  this  p-value,  there  is  insufficient 
evidenee  to  rejeet  the  null  hypothesis. 

This  ehapter  provided  an  in-depth  analysis  of  the  data,  to  inelude  the 
results  of  the  replieation  of  Elder’s  study.  Chapter  IV  presents  the  eonclusions  of  this 
study  and  provides  reeommendations  for  future  studies. 
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IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


A,  CONCLUSIONS 

The  ARJ  is  described  as  a  platform  where 

....  Acquisition  Corps  members  and  other  readers  from  government, 
Congress,  industry  and  academe  are  encouraged  to  use  the  Acquisition 
Review  Quarterly  (ARQ),  now  ARJ,  as  their  professional  forum  for 
discussion  and  exchange  of  policies,  research,  information,  and  opinion.'^^ 

The  research  conducted  in  both  studies  suggests  that  the  ARJ  is  accomplishing 
this  mission.  Acquisition  Reform  Initiatives  date  back  to  1961  and  are  still  occurring  to 
date;  this  continuing  in  reform  is  reflected  in  the  ARJ. 

The  ARJ  has  shown  some  distinctive  trends  based  on  the  analysis  described  in 
Chapters  II  and  III.  The  ARJ  is  still  a  new  journal;  therefore,  the  trends  are  a  good 
representation  of  a  growing  community.  The  ARJ  reflects  a  number  of  qualitative 
studies,  but  shows  progress  in  the  number  of  published  quantitative  studies.  The  trends  in 
the  types  of  research  performed,  research  design,  and  data  analysis  are  currently  in  line 
with  the  recommendations  of  the  research  community.  So  as  long  as  the  ARJ  continues 
to  publish  articles  of  its  current  caliber,  the  acquisition  community  should  only  expect  to 
see  more  quantitative  research  in  future  issues. 

The  research  shows  that  both  practitioners  and  academe  are  publishing  articles  in 
the  ARJ.  Practitioners’  contributions  are  slightly  greater  in  number  than  those  of 
academe,  but  the  difference  is  less  than  five  percent.  The  results  are  comparable  to  the 
mission  statement,  which  welcomes  “Acquisition  Corps  members  and  other  readers  from 
government.  Congress,  industry  and  academe,”43  which  leads  to  the  conclusion  that  the 
ARJ  is  capturing  its  intended  audience.  A  majority  of  the  articles  thus  far  have  been 
contributed  by  civilian  universities  and  military  organizations.  Organizations  such  as  the 
General  Accounting  Office  (GAO),  the  military  universities,  and  Canadian  forces 

Acquisition  Review  Quarterly,  Editorial  Mission,  Summer  1994,  no  page  number  noted, 
editorial  submission. 

^3  Acquisition  Review  Quarterly,  Editorial  Mission,  Summer  1994,  no  page  number  noted, 
editorial  submission. 
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(international)  were  more  appropriately  elassified  under  the  military  organizations 
umbrella.  However,  the  military  universities  are  trailing  civilian  organizations  by  less 
than  one  percent.  Again,  this  is  a  reflection  of  the  range  of  the  AllTs  audience  and  the 
fact  that  acquisition  is  of  interest  to  not  only  military  personnel,  but  to  the 
Acquisition  Corps. 

Since  1997,  the  number  of  authors  per  articles  has  increased.  It  has  moved  away 
from  just  one  and  two  authors  per  article  and  an  influx  of  three  and  four  authors  per 
articles  is  apparent.  This  shows  that  collaboration  is  encouraged  among  authors. 

Overall,  the  ARJ  is  meeting  its  published  mission  statement  and  in  the  upcoming 
issues  we  should  see  an  increase  in  articles  that  discuss  quality,  quality  assurance  (QA), 
and  configuration  management  because  they  have  been  topics  of  interest  to  the 
Acquisition  Corps. 

B,  LIMITATIONS  OF  THIS  STUDY 

Two  main  points  that  limited  the  reliability  of  this  type  of  study  are  the  human 
factors  and  the  methodology.  Human  factors  include  the  level  of  experience  and  interest 
of  the  researchers  and  the  limited  time  frame  of  the  study.  Having  done  this  for  the  first 
time,  the  quality  of  the  coded  judgments — despite  pleasantly  high  inter-coder 
agreement — was  not  entirely  dependable.  The  methodology  from  the  literature  reviews 
(article  reading)  to  the  coding  process  (categorical  assignments)  was  subject  to 
interpretations  as  well. 

C.  RECOMEND ATIONS  FOR  FUTURE  RESEARCH 

This  study  should  be  considered  a  stepping  stone  for  future  research  on  the  ARJ. 
“An  explicit  description  of  the  research  process  is  needed  in  order  to  increase  the 
reliability  of  the  research  in  question  so  as  to  render  it  possible  for  other  researchers  to 
replicate  the  research  and  its  findings. ”^4  Recommendations  include: 


^  Gyongyi  Kovacs  and  Karen  .M.  Spens,  “A  Content  Analysis  of  Research  Approaches  in  Logistics 
Research,”  International  Journal  of  Physical  Distribution  and  Logistics  Management,  Vol.  36,  No.  5,  2006, 
p.  385. 
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1,  Use  a  Group  of  Coders  to  Replicate  the  Study 

Two  important  threats  to  the  validity  of  qualitative  eonclusions  are  the  selection 
of  data  that  fit  the  researcher’s  existing  theory  or  preconceptions  and  the  selection  of  data 
that  “stands  out”  to  the  researcher .45  The  group  should  represent  a  good  sample  size  of 
the  Acquisition  Corps. 

2.  In-Depth  Study  of  the  Themes 

Having  read  Elder’s  perception  on  the  article  submission,  the  researchers  in  some 
ways,  agreed  with  him.  Elder  said,  “.  .  .  the  type  of  article  submitted  for  consideration  for 
publication  was  driven  primarily  by  potential  authors  aware  of  the  journal. ”46  In  the 
fourth  time  period,  the  ARJ  posted  specific  themes  intended  for  the  next  issue’s 
publication.  This  may  change  the  course  of  article  submission.  If  this  requirement  is 
maintained,  article  submission  will  be  filtered  in  every  issue. 

booking  back  to  the  definition  of  defense  acquisition  as  “the  conceptualization, 
initiation,  design,  development,  test,  contracting,  production,  deployment,  logistics 
support,  modifications,  and  disposal  of  weapons  systems  and  other  systems,  supplies,  or 
services  .  .  .”47  it  can  be  safely  assumed  that  these  were  adequately  represented.  As  this 
study  delved  into  the  subject  matter  areas,  it  seemed  that  minimal  research  was  devoted 
to  quality  and  QA. 

Eor  the  purpose  of  this  study,  one  theme  was  assigned  per  article;  however,  some 
articles  had  more  than  one  theme.  A  study  of  the  themes  will  provide  a  more 
comprehensive  list  of  the  themes  discussed  in  the  ARJ. 


45  Michael  Huberman  and  Matthew  B.  Miles,  Qualitative  Data  Analysis,  2nd  Edition, 
Thousand  Oaks,  CA,  1994,  p.  253. 

46  Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  33. 

47  Acquisition  Review  Quarterly,  “Guidelines  for  Authors,”  Winter  2002,  p.  85. 
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3,  Detailed  Study  of  the  Three  Categories 

Each  article  was  assigned  to  one  category  for  type  of  researeh  performed,  type  of 
researeh  design,  and  type  of  data  analysis,  when,  in  reality,  each  article  eould  have  more 
than  one  category  for  eaeh  of  the  three  areas.  In  order  to  better  understand  the  trends  it  is 
recommended  that  a  study  be  done  assigning  more  than  one  eategory  to  each  article  when 
applieable.  Although  more  extensive,  this  will  provide  a  elearer  picture  of  the  direction 
of  not  only  the  ARJ,  but  also  the  Aequisition  Community. 

Although  57  percent  of  Literature  and  Normative  reviews  about  the  eoneepts  of 
aequisition  were  already  submitted,  the  ARJ  still  needs  to  eontinue  this  trend,  since  this  is 
a  growing  publication  and  there  is  always  a  benefit  derived  from  the  knowledge  shared 
by  these  types  of  researeh. 

The  potential  for  different  interpretations  regarding  the  categories  exists.  In  2005, 
Karen  Spens  and  Gyongyi  Kovacs  recommended  some  measures  addressing  the  validity 
and  reliability  of  Content  Analysis.  The  following  are  taken  from  Table  I  of  their 
Content  Analysis  written  in  2006: 

•  Develop  clear  categorization  sehemes  and  decision  rules  for  categorization 

•  Follow  theoretical  framework  and  pre-define  eategories  for  each  variable 

•  Ensure  mutual  exclusiveness,  independence,  and  exhaustiveness 
of  eategories 

•  Fine  tune  of  eategory  development  during  the  coding  process 

•  Ensure  reliability  of  coding  instrument 

•  Ensure  the  reliability  of  the  coded  data  set  through  the  use  of  multiple  coders 

•  Assess  measurement  reliability 

•  Assess  eoding  consisteney  and  stability's 

Gyongyi  Kovacs  and  Karen  M.  Spens,  “A  Content  Analysis  of  Research  Approaches  in  Logistics 
Research,”  International  Journal  of  Physical  Distribution  and  Logistics  Management,  Vol.  36,  No.  5,  2006, 
pp.  380-381. 
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Reflecting  back  on  the  data  collection  and  coding  process  of  this  study,  the  two 
researchers  made  every  effort  to  clearly  establish  unanimous  interpretations  of  the 
categories.  The  regular  discussions  definitely  helped  to  clear  up  confusion;  however, 
there  was  always  the  potential  for  one  researcher  to  influence  the  other,  so  consistency 
was  not  established.  Since  each  article  is  not  necessarily  exclusive  to  only  one  set  of 
categories,  it  was  very  difficult  to  come  to  an  agreement.  Compromise  was  necessary  to 
assign  categories  to  the  articles. 

4,  Genealogy  of  Authors 

This  study  provided  different  information  about  the  contributing  authors:  their 
educational  background  and  professional  background  at  least.  It  showed  that  the  majority 
of  them  came  from  civilian  universities.  In  the  future,  research  should  be  undertaken  to 
study  the  valuable  contributors  in  the  acquisition  field  who  are  instrumental  to  the 
success  of  the  ARJ^'^  Such  a  study  will  lead  to  an  analysis  of  the  productivity  of 
contributing  authors  in  the  ARJ  mainly  to  show  that  the  field  is  growing  like  other 
related  journals. 

D,  SUMMARY 

This  study  has  proven  that  ARJ  has  met  its  editorial  mission  and  has  published 
articles  “in-sync”  with  the  Acquisition  Corps.  Theme  issues  are  the  future  of  ARJ  which 
could  change  the  current  trends  presented  in  this  study.  If  that  should  occur  then  the  ARJ 
should  revert  back  to  casual  models  to  increase  quantitative  studies. 


Jack  R.  Meredith  and  Kwasi  Amoako-Gyampah,  “The  Genealogy  of  Operations  Management,” 
Journal  of  Operations  Management,  Vol.  9,  No.  2,  pp.  146-167. 


49 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


50 


APPENDIX  A.  ARTICLE  #,  YEAR,  AUTHOR(S),  TITLE,  AND 
THEMES  OF  ARTICLES  INCLUDED  IN  THE  RESEARCH 


ART 

# 

YEAR  AUTHOR(S) 

TITLE 

THEMES 

1 

1 994  Preston,  Colleen 

Acquisition  Reform:  Making  it  a  Reality 

Acquisition  Reform 

2 

1994  LaBerge,  Walter  B. 

Restructuring  DoD:  Study  the  High-Tech  Commercial  World 

Acquisition  Reform 

3 

1994  Christensen,  David  S. 

Cost  Overrun  Optimism:  Fact  or  Fiction 

Cost,  Schedule, 
Performance 

4 

1994  Allen,  Gail;  Yoos  II, 
Charles 

Through  a  Glass  Darkly:  The  Anomaly  of  Streamlined 

Management 

Streamline  Acquisition 

5 

1994  Morrison  Jr.,  Robert 

Mobilizing  the  Defense  Contracting  Process 

Laws,  Policies, 
Regulations 

6 

1994  Chambers,  George 

Variance  Analysis  Within  C/SCSC  Programs 

Cost,  Schedule, 
Performance 

7 

1994  Lynn,  Larry 

The  Role  of  Demonstration  Approaches  in  Acquisition  Reform 

Acquisition  Reform 

8 

1 994  Horton,  Peter 

Converting  the  Military-Industrial  Complex:  Why  is  it  difficult 

Defense  Industry 

9 

1994  Templin,  Carl 

Defense  Contracting  Buyer-Seller  Relationships:  Theoretical 
Approaches 

Defense  Industry 

10 

1994  Shields,  John  T 

Factors  Affecting  New  Product  Development 

Procurement 

11 

1994  Pariseau,  Richard; 
Oswalt,  Ivar 

Using  Data  Types  and  Scales  for  Analysis  and  Decision  Making 

Decision  Making 

12 

1994  Greenburg,  Harvey  R.; 
Palley,  Lynn  B. 

Coming  Up  Golden:  Defense  Acquisition  Board  Review  Guide  for 
Program  Offices 

Program  Management 

13 

1994  Goodman,  Sherri 

The  Lemon  Juice  Solution:  Pollution  Prevention  and  Acquisition 
Reform 

Acquisition  Reform 

14 

1994  Neilson,  Robert 

The  Role  of  Information  Technology  in  National  Security  Policy 

Laws,  Policies, 
Regulations 

15 

1994  Shebalin,  Paul  V. 

Software  Development  Standards  and  the  DoD  Program  Manager 

Software 

16 

1994  Sutton,  Jeanne  C. 

Marrying  Commercial  and  Military  Technologies:  A  New  Strategy 
for  Maintaining  Technological  Superiority 

Technology 

17 

1 994  Williams,  Richard  M. 

Small  Business  Manufacturing:  An  Important  Component  of  the 

U.S.  Defense  Industrial  Base 

Technology 

18 

1 994  Glenn,  Mark  W. 

Using  Explicit  Demand  Curves  in  an  Acquisition  Strategy 

Contracting 

19 

1995  Byms  Jr,  Edward  V.; 
Corban,  J.  Eric;  Ingalls, 
Stephen  A. 

A  Novel  Cost-Benefit  Analysis  for  Evaluation  of  Complex  Military 
Systems 

Logistics 

20 

1995  Wells,  Rita  Lappin 

Contracting  Readiness:  Timely  Support  for  Military  Operations 

Contracting 

21 

1995  Scott,  William,  et  al. 

Project  Kaizen  Looks  at  Congressional  Oversight  of  Defense 
Acquisition  Programs 

Acquisition  Reform 

22 

1995  Rhoads,  Dusty 

Is  DAWIA  worth  it?  An  Approach  to  Analyzing  the  Impacts 

Education  and  Training 

23 

1995  Haimes,  Yacov; 
Chittister,  Clyde 

An  Acquisition  Process  for  the  Management  of  Non-Technical 

Risks  Associated  with  Software  Development 

Software 

24 

1995  Christensen,  David; 
Ferens,  Daniel 

Using  Earned  Value  for  Performance  Measurement  on  Software 
Development  Projects 

Software 

25 

1 995  Cancian,  Mark 

Acquisition  Reform:  It’s  not  as  easy  as  it  seems 

Acquisition  Reform 

26 

1995  Battershell,  A.  Lee 

Technology  Approach:  DoD  Versus  Boeing  (A  Comparative  Study) 

Technology 

27 

1995  Kwatnoski,  Richard 

Cooperative  Acquisition  Projects  in  the  Pacific  Rim 

International 

28 

1995  Dakin,  Timothy 

What  Every  Government  Employee  Should  Know  About 

Post-Federal  Employment  Restrictions 

Laws,  Policies,  and 
Regulations 

29 

1996  Doton,  Larry 

Integrating  Technology  to  Reduce  Fratricide 

Technology 

30 

1996  Schaller,  Michael  E. 

The  Impact  of  Technical  Data  Transfers  Problems  during  a 

Transition  of  Weapons  System  Production  between  Nations 

Technology 

31 

1 996  Cothran,  Julie 

Battle  Labs:  Tools  and  Scope 

Technology 

32 

1996  Wilson  Jr.,  John  R. 

Battle  Labs:  What  Are  They,  Where  Are  They  Going? 

Technology 

33 

1996  Hewitt,  Clyde 

Getting  to  the  On-Ramp  of  the  Information  Highway 

Technology 

34 

1996  LaBerge,  Walter  B. 

Cycle  Time:  A  Military  Imperative  As  Well 

Cost,  Schedule, 
Performance 

35 

1996  Luman,  Ronald  R.; 

Scotti,  Richard  S. 

The  System  Architect  Role  in  Acquisition  Program  Integrated 

Product  Teams 

Acquisition  Reform 

36 

1996  Snider,  Keith  F. 

DAWIA  and  the  Price  of  Professionalism 

Education 

37 

1996  Ferrara,  Joe 

DoD's  5000  Documents:  Evolution  and  Change  in  Defense 
Acquisition  Policy 

DoD  Framework 

38 

1996  D'Agostino,  Davi  M. 

Transatlantic  Cooperative  Weapons  Development:  How  can  we 
Better  Ensure  Success? 

Integrated  Product  and 
Process  Development/ 
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39  1996  Bregard,  Richard  W.;  Implementing  Integrated  Product  Development:  A  Project 

Chasteen,  Taylor  Manager’s  Perspective 

40  1997  Ciccotello,  Conrad  S.;  Rethinking  Twenty-First  Century  Acquisition:  Emerging  Trends  for 

Green,  Steve  G.;  Efficiency  Ends 

Homyak,  Martin 

41  1997  Nissen,  Mark  Reengineering  the  REP  Process  Through  Knowledge-Based 

Systems 

42  1997  Ranquet,  Robert  Think  Tanks  and  the  National  Security  Strategy  Formulation 

Process:  A  Comparison  of  Current  American  and  French  Patterns 

43  1997  Haimes,  Yacov;  A  Holistic  Management  Framework  for  Software  Acquisition 

Chittister,  Clyde; 

Schooff,  Richard 

44  1997  Steves,  Michael  R.  Addressing  Risk  Management  in  Non-Development  Items 

Acquisition  Programs 

45  1997  Pinker,  Aron,  Smith;  Selecting  Effective  Acquisition  Process  Metrics 

Charles  G.;  Booher,  Jack 
W. 

46  1997  Chew,  James  S.  B.  Commercial  Best  Practices  and  the  DoD  Acquisition  Process 

47  1997  O'Connor,  Michael  F.;  A  Decision  Support  Procedure  for  Best  Value  Source  Selections 

Paris,  Janine  L.; 

Lovelace,  Joan  S. 

48  1997  Peeler  Jr.,  David  L.  Applying  Dialectic  to  Acquisition  Strategy 

49  1997  Rush,  Benjamin  C.  Cost  as  an  Independent  Variable:  Concepts  and  Risks 

50  1997  Washington,  William  N.  A  Review  of  the  Literature:  Competition  Versus  Sole  Source 

Procurement 

51  1997  Anderson,  Timothy  P.;  Cost  Estimating  Risk  and  Cost  Estimating  Uncertainty  Guidelines 

Cherwonik,  Jeffrey  S. 

52  1997  Barrick,  Alan  E.;  Alberts,  Acquisition  of  State-of-the-Art  Logistics  Combat  Support  System: 

Henry  C.  The  Joint  Logistics  Advanced  Concept  Technology  Demonstration 

Program 

53  1997  Clark,  Rolf  The  Hidden  Implications  of  Force  Changes 

54  1997  Frank,  Deborah  A  Theoretical  Consideration  of  Acquisition  Reform 

55  1997  Garcia,  Andrea;  Keyner,  The  Defense  Acquisition  Workforce  Improvement  Act:  Five  Years 

Hugo;  Robillard,  Later 

Thomas;  VanMullekom, 

Mary 

56  1997  Harman,  Beryl  The  Government  Performance  Act:  Strategic  Planning  of  the  Future 

57  1997  Washington,  William  N.  Some  New  Approaches  to  “Reward  Contracting” 

58  1997  Wollover,  David  R.  Quality  Function  Deployment  as  a  Tool  for  Implementing  Cost  as 

an  Independent  Variable 

59  1997  Kankey,  Roland  D.;  Focused  Graduation  Education:  An  Invisible  but  Real  Competitive 

Muczyk,  Jan  P.;  Ely,  Neal  Edge 
M. 

60  1997  Brower,  Michael  J.  Outland:  The  Vogue  of  DoD  Outsourcing  and  Privatization 

61  1997  Hall,  Mary-Jo  Changing  the  Way  we  Assess  Leadership 

62  1997  Davis,  Randy;  Phillips,  The  Phoenix  Rises 

Bill;  Vazquez,  Bud 

64  1997  Bratt,  Gary  M.;  Estimating  the  Health  Hazard  Costs  of  Army  Materiel:  A  Method 

Doganiero,  Donna  M.;  for  Helping  Program  Managers  Make  Informed  Health  Risks 
Spencer,  Clark  0.  Decisions 

65  1998  Kock,  Ned  Government  Transformation  and  Structural  Rigidity:  Redesigning  a 

Service  Acquisition  Process 

66  1998  Crecine,  John  P.;  A  Multimedia  Systems  Approach  to  National  Security  Policy, 

Salomone,  Michael  D.  Decision  Making,  and  Intelligence  Support 

67  1998  Delano,  Kenneth  J.  Identifying  Factors  that  Contribute  to  Program  Success 

68  1998  Ignizio,  James  P.  Integrating  Cost,  Effectiveness,  and  Stability 

69  1998  Dobbins,  James  H.;  Summary  Research  Report  on  Critical  Success  Factors  in  Federal 

Donnelly,  Richard  G.  Government  Program  Management 

70  1998  Roberts,  Nancy  C.  Radical  Change  by  Entrepreneurial  Design 

71  1998  Reardon,  Kathleen  K.;  Leadership  Styles  for  the  Five  Stages  of  Radical  Change 

Reardon,  Kevin  J.;  Rowe, 


Integrated  Product 
Team  (IPPD/IPT) 
International 

Laws,  Policies, 
Regulations 

Contracting 

Laws,  Policies, 

Regulations 

Software 


Risk 

Streamline  Acquisition 


Streamline  Acquisition 
Contracting 


Streamline  Acquisition 
Cost,  Schedule, 
Performance 
Contracting 

Cost,  Schedule, 
Performance 
Acquisition  Reform 


Laws,  Policies, 
Regulations 
Acquisition  Reform 
Education  and  Training 


Laws,  Policies, 

Regulations 

Contracting 

Cost,  Schedule, 

Performance 

Education  and  Training 


Contracting 

Organizational 

Behavior 

IPPD/IPT 

Cost,  Schedule, 
Performance 

Change 

Technology 

Program  Management 
Cost,  Schedule, 
Performance 
Program  Management 

Change 

Organizational 

Behavior 


52 


Alan  J. 

72  1998  Hocevar,  Susan  Page;  Team-Based  Redesign  as  Large-Scale  Change:  Applying  Theory  to 

Owen,  Walter  E.  the  Implementation  of  Integrated  Product  Teams 

73  1998  Judith  Gebauer;  Carrie  Impact  of  the  Internet  on  Procurement 

Beam;  Arie  Segev 

74  1998  Scacchi,  Walt;  Boehm,  Virtual  Systems  Acquisition:  Approach  and  Transitions 

Barry 

75  1998  Hildebrandt,  Gregory  G.  The  Use  of  Performance  Incentives  in  DoD  Contracting 

76  1998  Holland,  Lauren  The  Weapons  Acquisition  Process:  The  Impediments  to  Radical 

Reform 

77  1998  Boyd,  James  A.  Concept  of  Operations  and  Implementation  Plan  for  Industry 

Integrated  Logistics  System  (I2LS) 

78  1998  Ferreiro,  Larrie  D.  Organizational  Trust  in  Naval  Ship  Design  Bureaus:  France,  Great 

Britain,  and  the  United  States 

79  1998  McNally,  William  P.  Will  Commercial  Specifications  Meet  our  Future  Air  Power  Needs? 

80  1998  Muczyk,  Jan  P.  Generating  Needed  Modernization  Funds:  Streamlining  the 

Bureaucracy  -  Not  Outsourcing  and  Privatizing  -  is  the  Best 
Solution 

81  1998  Wong,  Carolyn,  An  Approach  for  Efficiently  Managing  DoD  Research  and 

Steinberg,  Paul,  Horn,  Development  Portfolios 
Kenneth;  Axelband, 

Elliot 

82  1998  Childress,  Alan;  Larson,  A  Case  for  International  Cooperative  Acquisitions:  Lessons  from 

James  Developing  and  Executing  a  Section  27  “Quayle”  Authority 

Program 

83  1998  Scafati,  Anthony  A.  A  Case  Study  for  the  Systems  Approach  for  Developing  Curricula: 

“Don’t  Throw  out  the  Baby  with  the  Bath  Water’” 

84  1998  Christensen,  David  S.  The  Cost  and  Benefits  of  the  Earned  Value  Management  Process 

85  1999  Pollock,  Neal  International  Cooperative  Research  and  Development  Programs 

86  1999  Christensen,  David  S.;  The  Impact  Of  The  Packard  Commission’s  Recommendations  on 

Searle,  David  A.;  Reducing  Cost  Overruns  on  Defense  Acquisition  Contracts 

Vickery,  Caisse 

87  1999  Smyth,  Joseph,  Lt  Col  The  Impact  of  the  Buy  American  Act  on  Program  Managers 

88  1999  Parker,  Tony  Logistics  Test  And  Evaluation:  An  Overview 

89  1999  Christensen,  David  S.  Value  Cost  Management  Report  to  Evaluate  the  Contractor’s 

Estimate  at  Completion 

90  1999  Alford,  Lionel  D.,  Lt  Col  The  Problem  with  Aviation  COTS 

91  1999  Washington,  William  N.  Depot  Utilization  and  Commercialization 

92  1999  Ainsley,  Robert  J.;  DoD  and  the  Change  Paradigm:  Change  Agents  Versus  Established 

Riordan,  James  Service  Roles,  Missions  and  Cultures 

93  1999  Raymond,  Fred  Quantify  Risk  To  Manage  Cost  and  Schedule 

94  1999  Alford,  Lionel  D.,  Knarr,  General  Flight  Test  Theory  Applied  to  Aircraft  Modifications 

Robert  C. 

95  1999  Fullerton,  Richard  L.;  Acquisition  Reform  Theory  and  Experimental  Evidence  For 

Linster,  Bruce  G.;  Tournament  Sponsors 
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Stephen 
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105  1999  Hanratty,  Michael  J.;  Open  Systems  and  the  Systems  Engineering  Process 

Lightsey,  Robert  H.; 

Larson,  Arvid 

106  1999  Myers,  Margaret  An  Investment-Based  Approach  for  Managing  Software-Intensive 

System 


Change 

Technology 

Modeling  and 

Simulation 

Contracting 

Organizational 

Behavior 

Logistics 

Organizational 

Behavior 

Acquisition  Reform 
Cost,  Schedule, 
Performance 

Research, 

Development,  Testing, 
and  Evaluation 
(RDT&E) 

International 


Education  and  Training 

Cost,  Schedule, 
Performance 
International 
Cost,  Schedule, 
Performance 

Laws,  Policies, 
Regulations 
Logistics 
Cost,  Schedule, 
Performance 
Defense  Industry 
Industry 
Change 

Risk 

RDT&E 

Contracting 


Industry 

Contracting 

Procurement 

Technology 

Contracting 

Contracting 

Acquisition  Reform 
Acquisition  Reform 

Program  Management 

Systems  Engineering 
(SE) 

Software 


53 


107  1999  Washington,  William  N.  “Subcontracting”  as  a  Solution,  Not  a  Problem,  in  Outsourcing 
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APPENDIX  B.  RESEARCHERS’  ARTICLE  SAMPLING  FOR 

THE  THEMES 


Refer  to  Appendix  A  for  the  articles  used  in  this  sampling. 

Technology:  Battle  Labs:  Tools  and  Scope  (Article  #32):  This  is  a  tool  for  the  rapid  insertion  of 
new  technology  into  weapons  systems. 

Cost,  Schedule,  and  Performance:  The  Cost  and  Benefits  of  the  Earned  Value  Management 
Process  (Article  #84):  This  literature  review  talks  about  the  how  the  cost,  schedule,  and 
performance  using  a  work  breakdown  structure  help  in  the  management  process. 
Additionally,  it  talks  about  the  benefits  of  the  indices. 

Contracting:  The  Transformation  of  Contract  Incentive  Structures  (Article  #177):  An 

acquisition  and  operations  contract  combines  an  award  fee  and  success  fee  to  include  a  cost 
mitigation  approach.  The  incentive  program  asks  for  government  and  contractor  share  to 
succeed. 

Acquisition  Reform:  Towards  Centralized  Control  of  Defense  Acquisition  Programs 

(Article  #199):  The  new  acquisition  framework  has  more  phases  and  added 

more  reviews. 

Education  &  Training:  Engineering  Management  Training:  Comparing  Experiential  versus 
Lecture  Methods  of  Instruction  (Article  #121):  He  claims  that  the  experiential  method  of 
instructions  is  more  effective  in  learning  and  soliciting  student’s  reaction. 

Risk:  Understanding  Risk  Management  in  the  DoD  (Article  #171):  The  current  risk 

management  framework  does  not  adequately  emphasize  the  interface  between  risk 
management  and  contract  administration. 

Software:  Irreducible  Truths  of  Software-Intensive  Program  Management  (Article  #224):  It 
argues  that  software-intensive  program  do  not  automatically  make  a  program  manager 
achieve  the  goals  and  objectives  established  in  the  acquisition  program  baseline. 

Logistics:  Defining  and  Implementing  Performance-  Based  Logistics  in  Government 

(Article  #215):  This  article  talks  about  Performance-Based  Logistics  (PBL)’s 

implementation,  benefits,  and  infrastructure  changes  required. 

Laws,  Policies,  and  Regulations:  What  Every  Government  Employee  Should  Know  about  Post- 
Federal  Employee  Restrictions  (Article  #28):  The  article  talks  about  three  contradictory 
principal  laws  about  post-federal  employment  restrictions  on  acquisition  Government 
personnel. 

Organizational  Behavior:  Crisis  in  the  Acquisition  Workforce:  Some  Simple  Solutions 
(Article  #131):  The  author  here  suggests  some  approaches  to  deal  with  loss  due  to  retirement 
and  attrition  since  the  military  workforce  is  important  to  develop  weapons  systems. 

Lean  Implementation:  A  Person-Centered  Approach  to  Sustaining  a  Lean  Environment — Job 
Design  for  Self-Efficacy  (Article  #189):  Toyota  is  used  to  demonstrate  the  success  of  the  lean 
concept.  It  explores  relationships  between  corporate  beliefs,  employee  satisfaction,  and  self- 
efficacy. 

International:  International  Cooperative  Research  and  Development  Programs  (Article  #85): 
A  literature  review  on  International  Cooperative  R&D  Programs  gives  advice  on  how  to 
succeed  logistically  and  economically. 

Systems  Engineering:  Systems  Engineering  and  the  Joint  Strike  Fighter:  The  Flagship 
Program  for  Acquisition  Reform  (Article  #117):  The  article  shows  how  JSF  uses  Systems 
Engineering  process  as  an  acquisition  reform  initiative  to  achieve  its  warfighting  capability 
while  satisfying  the  needs  of  the  Navy,  Air  Force,  Marines,  and  the  United  Kingdom. 
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Research,  Development,  Test,  Evaluation:  General  Flight  Test  Theory  Applied  to  Aircraft 
Modifications  (Article  #94):  The  article  talks  about  the  use  of  fly-by- wire  technology  on  the 
C-130.  Small  modifications  need  extensive  testing  as  well. 

Management:  An  Assessment  of  Air  Force  Development  Portfolio  Management  Practices 
(Article  #147):  Decision  makers  need  to  weigh  benefits  and  cost  against  the  mission  needs 
for  different  weapons  systems  programs. 

Defense  Industry:  Consolidation  of  the  U.S.  Defense  Industrial  Base:  Impact  on  Research 
Expenditures  (Article  #148):  The  article  talks  about  the  merging  of  the  defense  contractors 
and  the  reduction  in  R&D  expenditures. 

Change:  Changing  the  Focus  of  Business  Process  Redesign  from  Activity  Flows  to  Information 
Flows:  A  Defense  Acquisition  Application  (Article  #132):  This  is  an  action  research  study  of 
a  business  process  redesign  that  supports  that  information  flow  instead  of  activity  flow  should 
the  focus  of  the  business  redesign  process. 

Streamline  Acquisition:  Commercial  Best  Practices  and  the  DoD  Acquisition  Process 
(Article  #46):  The  article  uses  the  best  practices  of  the  American  automobile  industry 
(Chrysler)  to  streamline  the  DoD  acquisition  process. 

Program  Management:  Identifying  Factors  that  Contribute  to  Program  Success  (Article  #67): 
Good  program  management  can  prevent  cost  overruns,  substandard  performance,  schedule 
delays,  and  also  cancellations  of  programs. 

Modeling  and  Simulation:  Building  a  Business  Case  for  Modeling  and  Simulation 
(Article  #111):  It  provides  a  framework  for  program  managers  within  DoD  to  know  when  to 
apply  M&S  to  programs. 

System  of  Systems:  Net-Centric  Warfare  and  its  Impact  on  Systems-of-Systems  (Article  #204): 
Air  Force  Distributed  Common  Ground  System  Block  10.2  is  fielding  systems  of  systems 
net-centric,  service- oriented  architecture  for  intelligence,  surveillance,  and  reconnaissance. 

DoD  Framework:  DoD's  5000  Documents:  Evolution  and  Change  in  Defense  Acquisition 
Policy  (Article  #37):  This  article  presents  a  historical  progression  of  the  DoD  5000  series  and 
documents. 

Decision  Making:  Using  Data  Types  and  Scales  for  Analysis  and  Decision  Making 
(Article  #11):  It  exposes  data  scales  and  numerical  manipulations  to  help  decision  makers 
choose  among  different  alternatives  and  make  decisions  on  the  allocation  of  scarce  resources. 

Procurement:  Purchasing  Performance:  A  public  Versus  Private  Sector  Comparison  of 
Commodity  Buying  (Article  #98):  The  article  compares  DoD  and  commercial  spending  and 
acquisition.  Careful  research  is  needed  to  benefit  from  the  purchasing  reform. 

Commercial  Industry:  Assessing  Industrial  Capabilities  for  Carbon  Fiber  Production 
(Article  #96):  Commercial  industries  play  an  important  role  in  the  production  and 
manufacturing  of  defense  aerospace  systems. 

Evolutionary  Acquisition:  Conflict  and  Ambiguity  Implementing  Evolutionary  Acquisition 
(Article  #164):  Evolutionary  acquisition  is  the  new  preferred  approach  in  acquiring  defense 
systems  as  the  Secretary  of  Defense  announced  in  2000.  This  article  features  the  struggles 
and  challenges  of  policy  implementation. 

IPPD/IPT:  The  Phoenix  Rises  (Article  #62):  This  article  shows  how  a  program  went  from 
possible  extinction  to  a  respective  model  based  on  the  ability  to  reduce  cost  and  schedule  yet 
still  come  out  ahead  through  the  use  of  integrated  product  teams  (IPT). 
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APPENDIX  C.  ELDER’S  THESIS  TOP  CONTRIBUTORS50 


Elder’s  Top  Contributors 


Authors 

Contributions 

Christensen,  David,  Dr. 

6* 

Nissen,  Mark  E.,  Dr. 

5 

Washington,  William 

5 

Alford,  Eionel  D.,  Lt  Col,  USAE 

4 

Arora,  Ashish 

3 

Besselman,  Joseph,  Maj,  USAF 

3 

Driessnack,  John  D.,  Et  Col,  USAF 

3 

Earkey,  Patrick 

3 

Einster,  Bruce  G. 

3 

Pollock,  Neal 

3 

Snider,  Keith  F.,  Dr. 

3 

Templin,  Carl,  Dr. 

3 

*Note:  This  includes  an  issue  introduction. 
Reproduced  from  Elder’s  thesis  (2005). 


Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  27. 
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APPENDIX  D.  ELDER’ S  TABLE  FOR  AUTHORS 

PER  ARTICLES  1 


Table  11.  Authors  per  Article 

One  Two  Three  Four 


Year  Mean  Author  Authors  Authors  Authors 


1 

1.16667 

83.3% 

16.7% 

0.0% 

0.0% 

2 

1.25 

72.7% 

18.2% 

9.1% 

0.0% 

3 

1.18182 

81.8% 

18.2% 

0.0% 

0.0% 

4 

1.76 

60.0% 

8.0% 

28.0% 

4.0% 

5 

1.66667 

57.1% 

23.8% 

14.3% 

4.8% 

6 

1.65217 

56.5% 

26.1% 

13.0% 

4.3% 

7 

1.68182 

68.2% 

9.1% 

9.1% 

13.6% 

8 

1.28571 

78.6% 

14.3% 

7.1% 

0.0% 

9 

2.21053 

31.6% 

26.3% 

31.6% 

10.5% 

10 

1.76471 

52.9% 

17.6% 

29.4% 

0.0% 

11 

2.15385 

23.1% 

38.5% 

38.5% 

0.0% 

SI  Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  28. 
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Frequencies 


APPENDIX  E.  ELDER’S  SUBJECT  (THEME)  CATEGORIES52 


Risk  Management 
Buyer/Seller  Relationships 
Performance  and  Measurement 
Organizational  Issues 
Research  and  Development 
Industry  Issues 
Contract  Management 
Policy  and  Regulation 
Program  Fielding  and  Implementation 
Analysis  and  Decision  Making 
Cost  and  Schedule 
Interoperability 
Management  and  Organizational  Behavior 
Acquisition  Strategy 
Reform  Initiatives 

0  5  10  15  20  25  30 


Subject  Categories 


^2  Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  19. 
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APPENDIX  F.  ELDER’S  FREQUENCY  AND  PROPORTION  OF 
SUBJECT  CATEGORY  PER  PERIOD53 


1 

2 

3 

1 

2 

3 

Reform  Initiatives 

9 

11 

7 

27 

Reform  Initiatives 

13.8% 

16.9% 

11.1% 

Acquisition  Strategy 

6 

9 

6 

21 

Acquisition  Strategy 

9.2% 

13.8% 

9.5% 

Management  and 

Management  and 

Organizational  Behavior 

3 

7 

10 

20 

Organizational  Behavior 

4.6% 

10.8% 

15.9% 

Interoperability 

4 

6 

10 

20 

Interoperability 

6.2% 

9.2% 

15.9% 

Cost  and  Schedule 

7 

7 

4 

18 

Cost  and  Schedule 

10.8% 

10.8% 

6.3% 

Analysis  and  Decision  Making 

6 

6 

1 

13 

Analysis  and  Decision  Making 

9.2% 

9.2% 

1.6% 

Program  Fielding  and 

Program  Fielding  and 

Implementation 

4 

5 

4 

13 

Implementation 

6.2% 

7.7% 

6.3% 

Policy  and  Regulation 

9 

2 

0 

11 

Policy  and  Regulation 

13.8% 

3.1% 

0.0% 

Contract  Management 

5 

2 

3 

10 

Contract  Management 

7.7% 

3.1% 

4.8% 

Industry  Issues 

2 

1 

6 

9 

Industry  Issues 

3.1% 

1.5% 

9.5% 

Research  and  Development 

3 

2 

2 

7 

Research  and  Development 

4.6% 

3.1% 

3.2% 

Organizational  Issues 

2 

3 

1 

6 

Organizational  Issues 

3.1% 

4.6% 

1.6% 

Performance  and 

Performance  and 

Measurement 

2 

2 

2 

6 

Measurement 

3.1% 

3.1% 

3.2% 

Buyen'Seller  Relationships 

2 

2 

2 

6 

Buyen'Seller  Relationships 

3.1% 

3.1% 

3.2% 

Risk  Management 

1 

0 

5 

6 

Risk  Management 

1.5% 

0.0% 

7.9% 

65 

65 

63 

^3  Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  20. 
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APPENDIX  G.  ELDER’S  PROPORTION  OF 
CONTRIBUTING  INSTITUTIONS54 


1 

2 

3 

Civilian  Universities 

14.1% 

24.0% 

28.6% 

DSMC.'DAU 

19.7% 

12.0% 

10.4% 

Civilian  Research  Firm.' 
Defense  Contractor 

9.9% 

12.0% 

19.5% 

Other  USAF 

1.4% 

16.0% 

10.4% 

Other  USA 

12.7% 

6.7% 

7.8% 

ICAF 

12.7% 

2.7% 

2.6% 

NPS 

4.2% 

8.0% 

3.9% 

AFIT 

9.9% 

2.7% 

2.6% 

SECDEF 

5.6% 

5.3% 

1.3% 

USAFA 

4.2% 

1 .3% 

3.9% 

Other  USN 

1.4% 

5.3% 

2.6% 

Other  DoD 

2.8% 

0.0% 

1.3% 

National  Defense  University 

1 .4% 

0.0% 

1.3% 

ACSC 

0.0% 

1 .3% 

1.3% 

Air  War  College 

0.0% 

1 .3% 

0.0% 

Other  Gov 

0.0% 

1 .3% 

0.0% 

USMA 

0.0% 

0.0% 

1.3% 

Naval  War  College 

0.0% 

0.0% 

1.3% 

^4  Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  29. 
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APPENDIX  H.  ELDER’S  PROPORTION  OF 
PRACTITIONERS  VS.  ACADEMICS55 


Table  14.  Pi  oportiou  of  Practitioners  vs.  Academics 


Contributor  Type 

1 

2 

3 

Practitioners 

66.2% 

53.3% 

55.8% 

Academics 

33.8% 

46.7% 

44.2% 

Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  30. 
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APPENDIX  I.  ELDER’S  PROPORTION  OF 
CIVILIAN  VS.  GOVERNMENT  ORGANIZATIONS56 


Table  13.  Proportion  of  Civilian  vs.  Govei  iiineiit  Orgaiiiziitious 


Contributor  Type 

1 

2 

3 

Civilian  Organizations 

23.9% 

36.0% 

48.1% 

Military  Organizations 

76.1% 

64.0% 

51.9% 

Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  p.  30. 
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APPENDIX  J.  ANOVA  TEST  FOR  THE  AUTHORS  PER 
ARTICLE  (233  ARTICLES) 


Anova:  Single  Factor 


SUMMARY 


Groups 

Count 

Sum 

Average 

Variance 

1994-1997 

64 

92 

1.4375 

0.59921 

1998-2000 

66 

109 

1.65152 

0.99977 

2001-2003 

50 

90 

1.8 

0.89796 

2004-2006 

53 

94 

1.77358 

0.67852 

ANOVA 

Source  of  Variation 

SS 

df 

MS 

F 

P-value 

F  crit 

Between  Groups 

4.82333 

3 

1.60778 

2.02277 

0.1115 

2,64402 

Within  Groups 

182.018 

229 

0.79484 

Total 

186.841 

232 

75 
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APPENDIX  K.  CHI-SQUARED  TEST  FOR  AUTHORS  PER 
ARTICLE  (233  ARTICLES) 


ACTUAL  FRE( 

QUENCY 

No.  of 

Authors/Article 

One 

Author 

Two 

Authors 

Three  + 
Authors 

Period  1  (1994-1997) 

46 

9 

9 

Period  2  (1998-2000) 

41 

13 

12 

Periods  (2001-2003) 

26 

10 

14 

Period  4  (2004-2006) 

23 

21 

9 

136 

53 

44 

EXPECTED  FR] 

EQUENCY 

No.  of 

Authors/Article 

One 

Author 

Two 

Authors 

Three  + 
Authors 

Period  1  (1994-1997) 

37 

15 

12 

Period  2  (1998-2000) 

39 

15 

12 

Periods  (2001-2003) 

29 

11 

9 

Period  4  (2004-2006) 

31 

12 

10 

136 

53 

43 
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APPENDIX  L.  ANOVA  TEST  FOR  THE  NUMBER  OF  AUTHORS 
PER  ARTICLE  (193  ARTICLES  TO  COMPARE  WITH 

ELDER’S  THESIS) 


Anova:  Single  Factor 


SUMMARY 


Groups 

Count 

Sum 

Average 

Variance 

Column  1 

64 

92 

1.4375 

0.59921 

Column  2 

66 

109 

1.65152 

0.99977 

Column  3 

63 

117 

1.85714 

0.8341 

ANOVA 

Source  of  Variation 

SS 

df 

MS 

F 

P-value 

F  crit 

Between  Groups 

5.59232 

2 

2.79616 

3.43977 

0.03408 

3.04347 

Within  Groups 

154.449 

190 

0.81289 

Total 

160.041 

192 
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APPENDIX  M.  ELDER’S  CATEGORIES57 


Article  Categories _ Elder  (2005) 


Type 

Period  1 

Period  2 

Period  3 

Normative 

44.6% 

46.2% 

33.3% 

Literature  Review 

3.1% 

1.5% 

3.2% 

Exploratory 

16.9% 

15.4% 

19.0% 

Methodology 

32.3% 

33.8% 

38.1% 

Hypothesis 

3.1% 

3.1% 

6.3% 

Research  Designs  _ Elder  (2005) 


Type 

Period  1 

Period  2 

Period  3 

Surveys 

1.5% 

10.8% 

4.8% 

Case  Study/ies 

13.8% 

13.8% 

22.2% 

Interviews 

6.2% 

3.1% 

6.3% 

Archival  Study 

29.2% 

21.5% 

22.2% 

Experiment 

0.0% 

3.1% 

3.2% 

Simulation 

N/A 

N/A 

N/A 

Topic  Presentation 

49.2% 

47.7% 

39.7% 

Mathematical  Model 

0.0% 

0.0% 

1.6% 

N/A  =  Not  Applicable. 

Article  Categories  _ Elder  (2005) 


Type 

Period  1 

Period  2 

Period  3 

Anecdotal  Evidence 

26.2% 

20.0% 

22.2% 

Statistical 

6.2% 

21.5% 

20.6% 

Content  Analysis 

15.4% 

7.7% 

11.1% 

Comparison  Analysis 

7.7% 

9.2% 

4.8% 

No  Analysis 

44.6% 

41.5% 

41.3% 

^7  Mitchell  Elder,  Capt,  USAF,  “An  Eleven  Year  Retrospective  of  the  Acquisition  Review  Journal,” 
Master’s  thesis.  Air  Force  Institute  of  Technology,  March  2005,  pp.  23-24,  26. 
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